Jul 16 21:47:31.263490 ip-10-0-128-255 systemd[1]: Starting Kubernetes Kubelet... Jul 16 21:47:31.770776 ip-10-0-128-255 kubenswrapper[2585]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 21:47:31.770776 ip-10-0-128-255 kubenswrapper[2585]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 16 21:47:31.770776 ip-10-0-128-255 kubenswrapper[2585]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 21:47:31.770776 ip-10-0-128-255 kubenswrapper[2585]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 16 21:47:31.770776 ip-10-0-128-255 kubenswrapper[2585]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 21:47:31.770776 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.651773 2585 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 16 21:47:31.770776 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656512 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 21:47:31.770776 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656526 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 21:47:31.770776 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656530 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 21:47:31.770776 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656533 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 21:47:31.770776 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656547 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 21:47:31.770776 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656550 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 21:47:31.770776 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656553 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 21:47:31.770776 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656556 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 21:47:31.770776 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656558 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 21:47:31.725052 ip-10-0-128-255 systemd[1]: Started Kubernetes Kubelet. Jul 16 21:47:31.773279 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656561 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 16 21:47:31.773279 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656564 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 21:47:31.773279 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656566 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 21:47:31.773279 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656569 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 21:47:31.773279 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656572 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 21:47:31.773279 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656575 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 21:47:31.773279 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656577 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 21:47:31.773279 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656580 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 21:47:31.773279 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656583 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 21:47:31.773279 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656585 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 21:47:31.773279 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656588 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 21:47:31.773279 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656591 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 21:47:31.773279 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656594 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 21:47:31.773279 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656597 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 21:47:31.773279 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656600 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 21:47:31.773279 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656602 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 21:47:31.773279 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656605 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 21:47:31.773279 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656608 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 21:47:31.773279 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656610 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 21:47:31.773279 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656613 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 21:47:31.775710 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656615 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 21:47:31.775710 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656618 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 21:47:31.775710 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656620 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 21:47:31.775710 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656623 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 21:47:31.775710 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656625 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 21:47:31.775710 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656628 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 21:47:31.775710 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656631 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 21:47:31.775710 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656635 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 21:47:31.775710 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656639 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 21:47:31.775710 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656642 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 21:47:31.775710 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656644 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 21:47:31.775710 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656647 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 21:47:31.775710 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656650 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 21:47:31.775710 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656653 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 21:47:31.775710 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656655 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 21:47:31.775710 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656658 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 21:47:31.775710 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656660 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 21:47:31.775710 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656663 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 21:47:31.775710 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656665 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 21:47:31.777674 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656668 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 21:47:31.777674 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656670 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 21:47:31.777674 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656674 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 21:47:31.777674 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656676 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 21:47:31.777674 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656679 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 21:47:31.777674 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656681 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 21:47:31.777674 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656683 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 21:47:31.777674 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656686 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 21:47:31.777674 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656690 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 21:47:31.777674 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656693 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 21:47:31.777674 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656696 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 21:47:31.777674 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656699 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 21:47:31.777674 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656701 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 21:47:31.777674 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656704 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 21:47:31.777674 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656707 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 21:47:31.777674 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656711 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 21:47:31.777674 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656715 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 21:47:31.777674 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656719 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 21:47:31.791657 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656722 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 21:47:31.791657 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656725 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 21:47:31.791657 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656727 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 21:47:31.791657 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656730 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 21:47:31.791657 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656733 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 21:47:31.791657 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656736 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 21:47:31.791657 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656739 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 21:47:31.791657 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656741 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 21:47:31.791657 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656744 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 21:47:31.791657 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656747 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 21:47:31.791657 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656749 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 21:47:31.791657 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656752 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 21:47:31.791657 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656755 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 21:47:31.791657 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656757 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 21:47:31.791657 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656761 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 21:47:31.791657 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656763 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 21:47:31.791657 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656766 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 21:47:31.791657 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656768 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 21:47:31.791657 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656771 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 21:47:31.791657 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656774 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 21:47:31.792799 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656777 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 21:47:31.792799 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656779 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 21:47:31.792799 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656782 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 21:47:31.792799 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656785 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 21:47:31.792799 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656787 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 21:47:31.792799 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656790 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 21:47:31.792799 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656792 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 21:47:31.792799 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656795 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 21:47:31.792799 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656797 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 21:47:31.792799 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656800 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 21:47:31.792799 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656802 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 21:47:31.792799 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656804 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 21:47:31.792799 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.656807 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 21:47:31.792799 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.657906 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 21:47:31.792799 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.657914 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 21:47:31.792799 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.657917 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 21:47:31.792799 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.657920 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 21:47:31.792799 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.657924 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 21:47:31.792799 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.657927 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 21:47:31.792799 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.657930 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 21:47:31.795083 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.657933 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 21:47:31.795083 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.657935 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 21:47:31.795083 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.657938 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 21:47:31.795083 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.657941 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 21:47:31.795083 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.657943 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 21:47:31.795083 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.657946 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 21:47:31.795083 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.657949 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 21:47:31.795083 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.657952 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 21:47:31.795083 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.657955 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 21:47:31.795083 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.657957 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 21:47:31.795083 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.657961 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 21:47:31.795083 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.657964 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 21:47:31.795083 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.657966 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 21:47:31.795083 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.657969 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 21:47:31.795083 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.657971 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 21:47:31.795083 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.657974 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 21:47:31.795083 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.657978 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 21:47:31.795083 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.657982 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 21:47:31.795083 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.657985 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 21:47:31.795926 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.657987 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 21:47:31.795926 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.657990 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 21:47:31.795926 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.657993 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 21:47:31.795926 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.657995 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 21:47:31.795926 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.657998 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 16 21:47:31.795926 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658000 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 21:47:31.795926 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658003 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 21:47:31.795926 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658006 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 21:47:31.795926 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658008 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 21:47:31.795926 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658011 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 21:47:31.795926 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658013 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 21:47:31.795926 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658015 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 21:47:31.795926 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658018 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 21:47:31.795926 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658021 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 21:47:31.795926 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658024 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 21:47:31.795926 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658027 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 21:47:31.795926 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658029 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 21:47:31.795926 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658032 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 21:47:31.795926 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658035 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 21:47:31.795926 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658038 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 21:47:31.797951 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658042 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 21:47:31.797951 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658047 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 21:47:31.797951 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658051 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 21:47:31.797951 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658056 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 21:47:31.797951 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658063 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 21:47:31.797951 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658068 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 21:47:31.797951 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658073 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 21:47:31.797951 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658078 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 21:47:31.797951 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658081 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 21:47:31.797951 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658085 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 21:47:31.797951 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658088 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 21:47:31.797951 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658091 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 21:47:31.797951 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658094 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 21:47:31.797951 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658098 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 21:47:31.797951 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658100 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 21:47:31.797951 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658104 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 21:47:31.797951 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658108 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 21:47:31.797951 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658111 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 21:47:31.797951 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658114 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 21:47:31.799661 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658117 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 21:47:31.799661 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658119 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 21:47:31.799661 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658122 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 21:47:31.799661 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658125 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 21:47:31.799661 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658127 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 21:47:31.799661 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658129 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 21:47:31.799661 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658132 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 21:47:31.799661 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658135 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 21:47:31.799661 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658137 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 21:47:31.799661 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658140 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 21:47:31.799661 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658143 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 21:47:31.799661 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658145 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 21:47:31.799661 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658148 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 21:47:31.799661 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658151 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 21:47:31.799661 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658154 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 21:47:31.799661 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658157 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 21:47:31.799661 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658159 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 21:47:31.799661 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658161 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 21:47:31.799661 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658165 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 21:47:31.799661 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658169 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 21:47:31.800583 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658171 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 21:47:31.800583 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658174 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 21:47:31.800583 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658177 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 21:47:31.800583 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658179 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 21:47:31.800583 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658182 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 21:47:31.800583 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658185 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 21:47:31.800583 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658188 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 21:47:31.800583 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658190 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 21:47:31.800583 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658193 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 21:47:31.800583 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658195 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 21:47:31.800583 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658198 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 21:47:31.800583 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658201 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 21:47:31.800583 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658203 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 21:47:31.800583 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658206 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 21:47:31.800583 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658608 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 21:47:31.800583 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658614 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 21:47:31.800583 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658617 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 21:47:31.800583 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658623 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 21:47:31.800583 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658626 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 21:47:31.800583 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658629 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 21:47:31.801419 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658632 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 21:47:31.801419 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658635 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 21:47:31.801419 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658637 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 21:47:31.801419 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658640 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 21:47:31.801419 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658643 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 21:47:31.801419 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658646 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 21:47:31.801419 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658649 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 21:47:31.801419 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658651 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 21:47:31.801419 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658654 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 21:47:31.801419 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658657 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 21:47:31.801419 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658665 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 21:47:31.801419 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658668 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 21:47:31.801419 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658670 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 21:47:31.801419 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658673 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 21:47:31.801419 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658675 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 21:47:31.801419 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658678 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 21:47:31.801419 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658680 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 21:47:31.801419 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658683 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 21:47:31.801419 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658685 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 21:47:31.802100 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658688 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 21:47:31.802100 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658691 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 21:47:31.802100 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658693 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 21:47:31.802100 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658696 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 21:47:31.802100 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658701 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 21:47:31.802100 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658703 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 21:47:31.802100 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658706 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 21:47:31.802100 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658708 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 21:47:31.802100 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658711 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 21:47:31.802100 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658713 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 21:47:31.802100 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658717 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 21:47:31.802100 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658720 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 21:47:31.802100 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658722 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 21:47:31.802100 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658725 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 21:47:31.802100 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658727 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 21:47:31.802100 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658730 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 21:47:31.802100 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658732 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 21:47:31.802100 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658735 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 21:47:31.802100 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658737 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 21:47:31.802100 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658740 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 21:47:31.802759 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658742 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 21:47:31.802759 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658745 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 21:47:31.802759 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658749 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 21:47:31.802759 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658753 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 21:47:31.802759 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658757 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 21:47:31.802759 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658759 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 21:47:31.802759 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658762 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 21:47:31.802759 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658765 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 21:47:31.802759 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658768 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 21:47:31.802759 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658770 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 21:47:31.802759 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658773 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 21:47:31.802759 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658776 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 21:47:31.802759 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658778 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 21:47:31.802759 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658781 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 21:47:31.802759 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658783 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 21:47:31.802759 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658786 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 21:47:31.802759 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658789 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 21:47:31.802759 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658792 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 21:47:31.802759 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658794 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 21:47:31.802759 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658797 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 21:47:31.803448 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658799 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 21:47:31.803448 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658801 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 21:47:31.803448 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658805 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 21:47:31.803448 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658808 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 21:47:31.803448 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658810 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 21:47:31.803448 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658813 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 21:47:31.803448 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658815 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 21:47:31.803448 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658817 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 21:47:31.803448 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658820 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 21:47:31.803448 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658822 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 21:47:31.803448 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658826 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 21:47:31.803448 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658831 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 21:47:31.803448 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658835 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 21:47:31.803448 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658837 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 21:47:31.803448 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658840 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 21:47:31.803448 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658842 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 21:47:31.803448 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658845 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 21:47:31.803448 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658848 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 16 21:47:31.803448 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658851 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 21:47:31.804064 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658854 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 21:47:31.804064 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658856 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 21:47:31.804064 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658858 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 21:47:31.804064 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658862 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 21:47:31.804064 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658865 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 21:47:31.804064 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658868 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 21:47:31.804064 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658871 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 21:47:31.804064 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658873 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 21:47:31.804064 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658876 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 21:47:31.804064 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658882 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 21:47:31.804064 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658884 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 21:47:31.804064 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658886 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 21:47:31.804064 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658889 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 21:47:31.804064 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658891 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 21:47:31.804064 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.658894 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 21:47:31.804064 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.658966 2585 flags.go:64] FLAG: --address="0.0.0.0" Jul 16 21:47:31.804064 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.658973 2585 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 16 21:47:31.804064 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.658978 2585 flags.go:64] FLAG: --anonymous-auth="true" Jul 16 21:47:31.804064 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.658983 2585 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 16 21:47:31.804064 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.658989 2585 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 16 21:47:31.804064 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.658994 2585 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 16 21:47:31.804907 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.658999 2585 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 16 21:47:31.804907 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659005 2585 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 16 21:47:31.804907 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659010 2585 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 16 21:47:31.804907 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659013 2585 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 16 21:47:31.804907 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659017 2585 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 16 21:47:31.804907 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659020 2585 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 16 21:47:31.804907 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659023 2585 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 16 21:47:31.804907 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659026 2585 flags.go:64] FLAG: --cgroup-root="" Jul 16 21:47:31.804907 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659029 2585 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 16 21:47:31.804907 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659032 2585 flags.go:64] FLAG: --client-ca-file="" Jul 16 21:47:31.804907 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659036 2585 flags.go:64] FLAG: --cloud-provider="external" Jul 16 21:47:31.804907 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659039 2585 flags.go:64] FLAG: --cluster-dns="[]" Jul 16 21:47:31.804907 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659043 2585 flags.go:64] FLAG: --cluster-domain="" Jul 16 21:47:31.804907 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659046 2585 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 16 21:47:31.804907 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659049 2585 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 16 21:47:31.804907 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659052 2585 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 16 21:47:31.804907 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659055 2585 flags.go:64] FLAG: --container-log-max-files="5" Jul 16 21:47:31.804907 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659058 2585 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 16 21:47:31.804907 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659062 2585 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 16 21:47:31.804907 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659065 2585 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 16 21:47:31.804907 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659069 2585 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 16 21:47:31.804907 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659072 2585 flags.go:64] FLAG: --contention-profiling="false" Jul 16 21:47:31.804907 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659075 2585 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 16 21:47:31.804907 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659078 2585 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 16 21:47:31.809078 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659081 2585 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 16 21:47:31.809078 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659084 2585 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 16 21:47:31.809078 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659089 2585 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 16 21:47:31.809078 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659092 2585 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 16 21:47:31.809078 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659095 2585 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 16 21:47:31.809078 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659098 2585 flags.go:64] FLAG: --enable-load-reader="false" Jul 16 21:47:31.809078 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659101 2585 flags.go:64] FLAG: --enable-server="true" Jul 16 21:47:31.809078 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659103 2585 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 16 21:47:31.809078 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659107 2585 flags.go:64] FLAG: --event-burst="100" Jul 16 21:47:31.809078 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659110 2585 flags.go:64] FLAG: --event-qps="50" Jul 16 21:47:31.809078 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659113 2585 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 16 21:47:31.809078 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659116 2585 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 16 21:47:31.809078 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659119 2585 flags.go:64] FLAG: --eviction-hard="" Jul 16 21:47:31.809078 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659122 2585 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 16 21:47:31.809078 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659125 2585 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 16 21:47:31.809078 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659128 2585 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 16 21:47:31.809078 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659131 2585 flags.go:64] FLAG: --eviction-soft="" Jul 16 21:47:31.809078 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659133 2585 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 16 21:47:31.809078 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659136 2585 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 16 21:47:31.809078 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659145 2585 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 16 21:47:31.809078 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659148 2585 flags.go:64] FLAG: --experimental-mounter-path="" Jul 16 21:47:31.809078 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659151 2585 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 16 21:47:31.809078 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659154 2585 flags.go:64] FLAG: --fail-swap-on="true" Jul 16 21:47:31.809078 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659157 2585 flags.go:64] FLAG: --feature-gates="" Jul 16 21:47:31.809078 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659160 2585 flags.go:64] FLAG: --file-check-frequency="20s" Jul 16 21:47:31.810066 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659163 2585 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 16 21:47:31.810066 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659166 2585 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 16 21:47:31.810066 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659169 2585 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 16 21:47:31.810066 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659174 2585 flags.go:64] FLAG: --healthz-port="10248" Jul 16 21:47:31.810066 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659177 2585 flags.go:64] FLAG: --help="false" Jul 16 21:47:31.810066 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659180 2585 flags.go:64] FLAG: --hostname-override="ip-10-0-128-255.ec2.internal" Jul 16 21:47:31.810066 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659183 2585 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 16 21:47:31.810066 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659186 2585 flags.go:64] FLAG: --http-check-frequency="20s" Jul 16 21:47:31.810066 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659188 2585 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 16 21:47:31.810066 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659193 2585 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 16 21:47:31.810066 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659197 2585 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 16 21:47:31.810066 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659199 2585 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 16 21:47:31.810066 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659202 2585 flags.go:64] FLAG: --image-service-endpoint="" Jul 16 21:47:31.810066 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659205 2585 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 16 21:47:31.810066 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659208 2585 flags.go:64] FLAG: --kube-api-burst="100" Jul 16 21:47:31.810066 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659211 2585 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 16 21:47:31.810066 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659229 2585 flags.go:64] FLAG: --kube-api-qps="50" Jul 16 21:47:31.810066 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659234 2585 flags.go:64] FLAG: --kube-reserved="" Jul 16 21:47:31.810066 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659238 2585 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 16 21:47:31.810066 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659241 2585 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 16 21:47:31.810066 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659244 2585 flags.go:64] FLAG: --kubelet-cgroups="" Jul 16 21:47:31.810066 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659247 2585 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 16 21:47:31.810066 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659250 2585 flags.go:64] FLAG: --lock-file="" Jul 16 21:47:31.810066 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659252 2585 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 16 21:47:31.818806 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659255 2585 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 16 21:47:31.818806 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659258 2585 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 16 21:47:31.818806 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659263 2585 flags.go:64] FLAG: --log-json-split-stream="false" Jul 16 21:47:31.818806 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659272 2585 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 16 21:47:31.818806 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659275 2585 flags.go:64] FLAG: --log-text-split-stream="false" Jul 16 21:47:31.818806 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659278 2585 flags.go:64] FLAG: --logging-format="text" Jul 16 21:47:31.818806 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659281 2585 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 16 21:47:31.818806 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659284 2585 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 16 21:47:31.818806 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659287 2585 flags.go:64] FLAG: --manifest-url="" Jul 16 21:47:31.818806 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659290 2585 flags.go:64] FLAG: --manifest-url-header="" Jul 16 21:47:31.818806 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659294 2585 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 16 21:47:31.818806 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659299 2585 flags.go:64] FLAG: --max-open-files="1000000" Jul 16 21:47:31.818806 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659302 2585 flags.go:64] FLAG: --max-pods="110" Jul 16 21:47:31.818806 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659305 2585 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 16 21:47:31.818806 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659308 2585 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 16 21:47:31.818806 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659311 2585 flags.go:64] FLAG: --memory-manager-policy="None" Jul 16 21:47:31.818806 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659314 2585 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 16 21:47:31.818806 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659318 2585 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 16 21:47:31.818806 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659321 2585 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 16 21:47:31.818806 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659324 2585 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 16 21:47:31.818806 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659331 2585 flags.go:64] FLAG: --node-status-max-images="50" Jul 16 21:47:31.818806 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659334 2585 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 16 21:47:31.818806 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659337 2585 flags.go:64] FLAG: --oom-score-adj="-999" Jul 16 21:47:31.818806 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659340 2585 flags.go:64] FLAG: --pod-cidr="" Jul 16 21:47:31.832253 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659343 2585 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 16 21:47:31.832253 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659349 2585 flags.go:64] FLAG: --pod-manifest-path="" Jul 16 21:47:31.832253 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659351 2585 flags.go:64] FLAG: --pod-max-pids="-1" Jul 16 21:47:31.832253 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659354 2585 flags.go:64] FLAG: --pods-per-core="0" Jul 16 21:47:31.832253 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659357 2585 flags.go:64] FLAG: --port="10250" Jul 16 21:47:31.832253 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659360 2585 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 16 21:47:31.832253 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659363 2585 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d24db276cb7db029" Jul 16 21:47:31.832253 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659366 2585 flags.go:64] FLAG: --qos-reserved="" Jul 16 21:47:31.832253 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659369 2585 flags.go:64] FLAG: --read-only-port="10255" Jul 16 21:47:31.832253 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659372 2585 flags.go:64] FLAG: --register-node="true" Jul 16 21:47:31.832253 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659374 2585 flags.go:64] FLAG: --register-with-taints="" Jul 16 21:47:31.832253 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659378 2585 flags.go:64] FLAG: --registry-burst="10" Jul 16 21:47:31.832253 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659387 2585 flags.go:64] FLAG: --registry-qps="5" Jul 16 21:47:31.832253 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659390 2585 flags.go:64] FLAG: --reserved-cpus="" Jul 16 21:47:31.832253 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659393 2585 flags.go:64] FLAG: --reserved-memory="" Jul 16 21:47:31.832253 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659396 2585 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 16 21:47:31.832253 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659399 2585 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 16 21:47:31.832253 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659402 2585 flags.go:64] FLAG: --rotate-certificates="false" Jul 16 21:47:31.832253 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659405 2585 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 16 21:47:31.832253 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659408 2585 flags.go:64] FLAG: --runonce="false" Jul 16 21:47:31.832253 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659413 2585 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 16 21:47:31.832253 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659416 2585 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 16 21:47:31.832253 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659418 2585 flags.go:64] FLAG: --seccomp-default="false" Jul 16 21:47:31.832253 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659421 2585 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 16 21:47:31.832253 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659424 2585 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 16 21:47:31.833101 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659429 2585 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 16 21:47:31.833101 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659432 2585 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 16 21:47:31.833101 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659435 2585 flags.go:64] FLAG: --storage-driver-password="root" Jul 16 21:47:31.833101 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659437 2585 flags.go:64] FLAG: --storage-driver-secure="false" Jul 16 21:47:31.833101 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659440 2585 flags.go:64] FLAG: --storage-driver-table="stats" Jul 16 21:47:31.833101 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659443 2585 flags.go:64] FLAG: --storage-driver-user="root" Jul 16 21:47:31.833101 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659445 2585 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 16 21:47:31.833101 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659449 2585 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 16 21:47:31.833101 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659451 2585 flags.go:64] FLAG: --system-cgroups="" Jul 16 21:47:31.833101 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659454 2585 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 16 21:47:31.833101 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659459 2585 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 16 21:47:31.833101 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659462 2585 flags.go:64] FLAG: --tls-cert-file="" Jul 16 21:47:31.833101 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659465 2585 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 16 21:47:31.833101 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659472 2585 flags.go:64] FLAG: --tls-min-version="" Jul 16 21:47:31.833101 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659475 2585 flags.go:64] FLAG: --tls-private-key-file="" Jul 16 21:47:31.833101 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659477 2585 flags.go:64] FLAG: --topology-manager-policy="none" Jul 16 21:47:31.833101 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659480 2585 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 16 21:47:31.833101 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659483 2585 flags.go:64] FLAG: --topology-manager-scope="container" Jul 16 21:47:31.833101 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659486 2585 flags.go:64] FLAG: --v="2" Jul 16 21:47:31.833101 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659489 2585 flags.go:64] FLAG: --version="false" Jul 16 21:47:31.833101 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659493 2585 flags.go:64] FLAG: --vmodule="" Jul 16 21:47:31.833101 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659497 2585 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 16 21:47:31.833101 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.659500 2585 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 16 21:47:31.833101 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659609 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 21:47:31.833101 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659613 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 21:47:31.833942 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659616 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 21:47:31.833942 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659618 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 21:47:31.833942 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659623 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 21:47:31.833942 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659626 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 21:47:31.833942 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659628 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 21:47:31.833942 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659631 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 21:47:31.833942 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659633 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 21:47:31.833942 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659638 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 21:47:31.833942 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659641 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 21:47:31.833942 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659643 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 21:47:31.833942 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659646 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 21:47:31.833942 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659648 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 21:47:31.833942 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659651 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 21:47:31.833942 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659653 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 21:47:31.833942 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659656 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 21:47:31.833942 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659658 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 21:47:31.833942 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659662 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 21:47:31.833942 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659666 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 21:47:31.833942 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659670 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 21:47:31.834577 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659672 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 21:47:31.834577 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659675 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 21:47:31.834577 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659677 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 21:47:31.834577 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659680 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 21:47:31.834577 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659683 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 21:47:31.834577 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659686 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 21:47:31.834577 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659688 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 21:47:31.834577 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659691 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 21:47:31.834577 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659693 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 21:47:31.834577 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659697 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 21:47:31.834577 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659699 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 21:47:31.834577 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659702 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 21:47:31.834577 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659704 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 21:47:31.834577 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659707 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 21:47:31.834577 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659709 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 21:47:31.834577 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659713 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 21:47:31.834577 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659716 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 21:47:31.834577 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659718 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 21:47:31.834577 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659721 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 21:47:31.834577 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659723 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 21:47:31.835237 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659727 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 21:47:31.835237 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659730 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 21:47:31.835237 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659733 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 21:47:31.835237 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659735 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 21:47:31.835237 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659738 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 21:47:31.835237 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659741 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 21:47:31.835237 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659743 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 21:47:31.835237 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659746 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 21:47:31.835237 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659749 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 21:47:31.835237 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659751 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 21:47:31.835237 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659754 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 21:47:31.835237 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659756 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 21:47:31.835237 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659759 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 21:47:31.835237 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659761 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 21:47:31.835237 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659764 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 21:47:31.835237 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659766 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 21:47:31.835237 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659769 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 21:47:31.835237 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659771 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 21:47:31.835237 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659774 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 21:47:31.835237 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659777 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 21:47:31.835237 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659779 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 21:47:31.835854 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659782 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 21:47:31.835854 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659785 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 21:47:31.835854 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659787 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 21:47:31.835854 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659790 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 21:47:31.835854 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659792 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 21:47:31.835854 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659795 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 21:47:31.835854 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659799 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 21:47:31.835854 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659801 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 21:47:31.835854 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659804 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 16 21:47:31.835854 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659806 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 21:47:31.835854 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659809 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 21:47:31.835854 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659812 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 21:47:31.835854 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659815 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 21:47:31.835854 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659818 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 21:47:31.835854 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659820 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 21:47:31.835854 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659822 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 21:47:31.835854 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659825 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 21:47:31.835854 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659827 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 21:47:31.835854 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659830 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 21:47:31.835854 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659832 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 21:47:31.836496 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659835 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 21:47:31.836496 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659837 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 21:47:31.836496 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659840 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 21:47:31.836496 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659842 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 21:47:31.836496 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659845 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 21:47:31.836496 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659849 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 21:47:31.836496 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659852 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 21:47:31.836496 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659855 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 21:47:31.836496 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659858 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 21:47:31.836496 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659861 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 21:47:31.836496 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659863 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 21:47:31.836496 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659866 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 21:47:31.836496 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659869 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 21:47:31.836496 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659873 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 21:47:31.836496 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659876 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 21:47:31.836496 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659879 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 21:47:31.836496 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.659882 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 21:47:31.837140 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.660488 2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 21:47:31.837140 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.666762 2585 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 16 21:47:31.837140 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.666777 2585 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 16 21:47:31.837140 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.666796 2585 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 16 21:47:31.837140 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.666802 2585 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 16 21:47:31.837140 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666856 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 21:47:31.837140 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666863 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 21:47:31.837140 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666866 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 21:47:31.837140 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666869 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 21:47:31.837140 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666872 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 21:47:31.837140 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666875 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 21:47:31.837140 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666878 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 21:47:31.837140 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666881 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 21:47:31.837140 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666884 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 21:47:31.837140 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666886 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 21:47:31.837787 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666889 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 21:47:31.837787 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666891 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 21:47:31.837787 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666894 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 21:47:31.837787 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666896 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 21:47:31.837787 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666899 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 21:47:31.837787 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666903 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 21:47:31.837787 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666907 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 21:47:31.837787 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666909 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 21:47:31.837787 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666912 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 21:47:31.837787 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666914 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 21:47:31.837787 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666917 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 21:47:31.837787 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666920 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 21:47:31.837787 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666923 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 21:47:31.837787 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666925 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 21:47:31.837787 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666928 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 21:47:31.837787 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666930 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 21:47:31.837787 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666933 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 21:47:31.837787 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666936 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 21:47:31.837787 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666940 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 21:47:31.838430 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666943 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 21:47:31.838430 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666946 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 21:47:31.838430 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666949 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 21:47:31.838430 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666951 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 21:47:31.838430 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666954 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 21:47:31.838430 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666956 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 21:47:31.838430 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666959 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 21:47:31.838430 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666962 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 21:47:31.838430 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666964 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 21:47:31.838430 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666970 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 21:47:31.838430 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666972 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 21:47:31.838430 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666975 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 21:47:31.838430 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666977 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 21:47:31.838430 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666980 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 21:47:31.838430 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666983 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 21:47:31.838430 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666985 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 21:47:31.838430 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666988 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 21:47:31.838430 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666991 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 21:47:31.838430 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666993 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 21:47:31.838430 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666996 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 21:47:31.839080 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.666999 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 21:47:31.839080 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667001 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 21:47:31.839080 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667004 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 21:47:31.839080 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667006 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 21:47:31.839080 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667009 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 21:47:31.839080 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667011 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 21:47:31.839080 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667014 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 21:47:31.839080 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667016 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 21:47:31.839080 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667019 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 21:47:31.839080 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667021 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 21:47:31.839080 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667024 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 21:47:31.839080 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667027 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 21:47:31.839080 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667030 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 21:47:31.839080 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667033 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 21:47:31.839080 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667035 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 21:47:31.839080 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667038 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 21:47:31.839080 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667041 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 21:47:31.839080 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667043 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 21:47:31.839080 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667046 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 21:47:31.839080 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667048 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 21:47:31.839697 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667051 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 21:47:31.839697 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667055 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 21:47:31.839697 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667058 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 21:47:31.839697 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667062 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 21:47:31.839697 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667065 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 21:47:31.839697 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667069 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 21:47:31.839697 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667072 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 21:47:31.839697 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667075 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 21:47:31.839697 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667077 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 21:47:31.839697 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667080 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 16 21:47:31.839697 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667083 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 21:47:31.839697 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667085 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 21:47:31.839697 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667088 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 21:47:31.839697 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667090 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 21:47:31.839697 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667092 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 21:47:31.839697 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667095 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 21:47:31.839697 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667099 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 21:47:31.839697 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667102 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 21:47:31.839697 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667104 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 21:47:31.839697 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667107 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 21:47:31.840336 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667110 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 21:47:31.840336 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667113 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 21:47:31.840336 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667115 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 21:47:31.840336 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667118 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 21:47:31.840336 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667120 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 21:47:31.840336 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667124 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 21:47:31.840336 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667128 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 21:47:31.840336 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667130 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 21:47:31.840336 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667133 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 21:47:31.840336 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667136 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 21:47:31.840336 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.667141 2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 21:47:31.840336 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667268 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 21:47:31.840336 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667275 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 21:47:31.840336 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667278 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 21:47:31.840759 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667283 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 21:47:31.840759 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667286 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 21:47:31.840759 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667289 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 21:47:31.840759 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667292 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 21:47:31.840759 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667294 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 21:47:31.840759 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667297 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 21:47:31.840759 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667300 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 21:47:31.840759 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667304 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 21:47:31.840759 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667307 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 21:47:31.840759 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667310 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 21:47:31.840759 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667313 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 21:47:31.840759 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667315 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 21:47:31.840759 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667318 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 21:47:31.840759 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667320 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 21:47:31.840759 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667324 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 21:47:31.840759 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667327 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 21:47:31.840759 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667329 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 21:47:31.840759 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667331 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 21:47:31.840759 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667334 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 21:47:31.841398 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667339 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 21:47:31.841398 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667342 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 21:47:31.841398 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667345 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 21:47:31.841398 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667348 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 21:47:31.841398 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667351 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 21:47:31.841398 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667353 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 21:47:31.841398 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667356 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 21:47:31.841398 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667359 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 16 21:47:31.841398 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667361 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 21:47:31.841398 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667364 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 21:47:31.841398 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667367 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 21:47:31.841398 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667369 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 21:47:31.841398 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667371 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 21:47:31.841398 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667376 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 21:47:31.841398 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667379 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 21:47:31.841398 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667381 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 21:47:31.841398 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667384 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 21:47:31.841398 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667386 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 21:47:31.841398 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667389 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 21:47:31.842022 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667391 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 21:47:31.842022 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667394 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 21:47:31.842022 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667396 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 21:47:31.842022 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667398 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 21:47:31.842022 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667401 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 21:47:31.842022 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667403 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 21:47:31.842022 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667406 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 21:47:31.842022 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667409 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 21:47:31.842022 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667415 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 21:47:31.842022 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667417 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 21:47:31.842022 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667420 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 21:47:31.842022 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667423 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 21:47:31.842022 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667425 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 21:47:31.842022 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667428 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 21:47:31.842022 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667430 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 21:47:31.842022 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667433 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 21:47:31.842022 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667435 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 21:47:31.842022 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667438 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 21:47:31.842022 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667440 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 21:47:31.842665 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667443 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 21:47:31.842665 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667446 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 21:47:31.842665 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667449 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 21:47:31.842665 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667452 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 21:47:31.842665 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667454 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 21:47:31.842665 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667457 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 21:47:31.842665 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667460 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 21:47:31.842665 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667463 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 21:47:31.842665 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667466 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 21:47:31.842665 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667468 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 21:47:31.842665 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667471 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 21:47:31.842665 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667473 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 21:47:31.842665 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667476 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 21:47:31.842665 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667478 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 21:47:31.842665 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667481 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 21:47:31.842665 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667483 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 21:47:31.842665 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667486 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 21:47:31.842665 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667488 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 21:47:31.842665 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667491 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 21:47:31.842665 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667494 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 21:47:31.843518 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667496 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 21:47:31.843518 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667499 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 21:47:31.843518 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667502 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 21:47:31.843518 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667504 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 21:47:31.843518 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667507 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 21:47:31.843518 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667509 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 21:47:31.843518 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667512 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 21:47:31.843518 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667514 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 21:47:31.843518 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667517 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 21:47:31.843518 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667519 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 21:47:31.843518 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667522 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 21:47:31.843518 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667524 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 21:47:31.843518 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667527 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 21:47:31.843518 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667529 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 21:47:31.843518 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667532 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 21:47:31.843518 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667535 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 21:47:31.843518 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667537 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 21:47:31.843518 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667540 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 21:47:31.843518 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:31.667542 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 21:47:31.844181 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.667548 2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 21:47:31.844181 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.667640 2585 server.go:962] "Client rotation is on, will bootstrap in background" Jul 16 21:47:31.844181 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.669678 2585 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 16 21:47:31.844181 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.671198 2585 server.go:1019] "Starting client certificate rotation" Jul 16 21:47:31.844181 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.671306 2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 21:47:31.844181 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.671345 2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 21:47:31.844181 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.673535 2585 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 21:47:31.844181 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.678400 2585 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 21:47:31.844181 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.691812 2585 log.go:25] "Validated CRI v1 runtime API" Jul 16 21:47:31.844181 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.693915 2585 log.go:25] "Validated CRI v1 image API" Jul 16 21:47:31.844181 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.695503 2585 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 16 21:47:31.844181 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.698083 2585 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c60be83f-7006-47b8-a88c-2d1bb0683d67:/dev/nvme0n1p3 dd06b237-a10a-4b6d-aca7-57d40ed55570:/dev/nvme0n1p4] Jul 16 21:47:31.907252 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.698097 2585 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 16 21:47:31.907252 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.698386 2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 21:47:31.913589 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.703550 2585 manager.go:216] Machine: {Timestamp:2026-07-16 21:47:31.702025568 +0000 UTC m=+0.324468513 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098973 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec257d731708b4df23d99f14d4508081 SystemUUID:ec257d73-1708-b4df-23d9-9f14d4508081 BootID:6faf412b-0ece-455c-9586-2c71096addb3 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:15:32:5c:6a:03 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:15:32:5c:6a:03 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:26:62:ec:4a:07:9a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 16 21:47:31.913589 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.703644 2585 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 16 21:47:31.913589 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.703719 2585 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 16 21:47:31.913589 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.707111 2585 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 16 21:47:31.913589 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.707140 2585 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-255.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 16 21:47:31.913589 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.707832 2585 topology_manager.go:138] "Creating topology manager with none policy" Jul 16 21:47:31.913589 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.707842 2585 container_manager_linux.go:309] "Creating device plugin manager" Jul 16 21:47:31.913589 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.707849 2585 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 21:47:31.913589 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.708464 2585 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 21:47:31.913589 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.709724 2585 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 16 21:47:31.913589 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.712494 2585 state_mem.go:36] "Initialized new in-memory state store" Jul 16 21:47:31.913589 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.712628 2585 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 16 21:47:31.913589 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.712657 2585 kubelet.go:486] "Attempting to sync node with API server" Jul 16 21:47:31.913589 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.712666 2585 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 16 21:47:31.913589 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.712679 2585 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 16 21:47:31.913589 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.712689 2585 kubelet.go:398] "Adding apiserver pod source" Jul 16 21:47:31.913589 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.712698 2585 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 16 21:47:31.913589 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.713054 2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 21:47:31.914242 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.713072 2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 21:47:31.914242 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.715173 2585 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 16 21:47:31.914242 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.716756 2585 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 16 21:47:31.914242 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.716798 2585 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 16 21:47:31.914242 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.717432 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 16 21:47:31.914242 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.717446 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 16 21:47:31.914242 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.717454 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 16 21:47:31.914242 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.717464 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 16 21:47:31.914242 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.717478 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 16 21:47:31.914242 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.717485 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 16 21:47:31.914242 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.717494 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 16 21:47:31.914242 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.717500 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 16 21:47:31.914242 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.717506 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 16 21:47:31.914242 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.717512 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 16 21:47:31.914242 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.717525 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 16 21:47:31.914242 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.717530 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 16 21:47:31.914242 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.718197 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 16 21:47:31.914242 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.718203 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 16 21:47:31.914242 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.721682 2585 server.go:1268] "Started kubelet" Jul 16 21:47:31.914242 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.721777 2585 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 16 21:47:31.914242 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.721835 2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-255.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 16 21:47:31.914242 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.721838 2585 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 16 21:47:31.914242 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.724776 2585 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 16 21:47:31.914242 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:31.725244 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-255.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 16 21:47:31.914242 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:31.725302 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 16 21:47:31.914242 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.726209 2585 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 16 21:47:31.914242 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.726456 2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zh9z2" Jul 16 21:47:31.914242 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.727378 2585 server.go:310] "Adding debug handlers to kubelet server" Jul 16 21:47:31.914242 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:31.728375 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-255.ec2.internal.18c2e36b257653c8 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-255.ec2.internal,UID:ip-10-0-128-255.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-255.ec2.internal,},FirstTimestamp:2026-07-16 21:47:31.72165524 +0000 UTC m=+0.344098185,LastTimestamp:2026-07-16 21:47:31.72165524 +0000 UTC m=+0.344098185,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-255.ec2.internal,}" Jul 16 21:47:31.914242 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:31.730157 2585 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 16 21:47:31.914242 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.732797 2585 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 16 21:47:31.914242 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.732808 2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 16 21:47:31.914242 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:31.732892 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-255.ec2.internal\" not found" Jul 16 21:47:31.914242 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.732930 2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zh9z2" Jul 16 21:47:31.914242 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.733322 2585 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 16 21:47:31.914242 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.733327 2585 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 16 21:47:31.914242 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.733342 2585 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 16 21:47:31.914242 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.733444 2585 reconstruct.go:97] "Volume reconstruction finished" Jul 16 21:47:31.914242 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.733452 2585 reconciler.go:29] "Reconciler: start to sync state" Jul 16 21:47:31.914242 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.733946 2585 factory.go:55] Registering systemd factory Jul 16 21:47:31.915315 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.733959 2585 factory.go:223] Registration of the systemd container factory successfully Jul 16 21:47:31.915315 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.734128 2585 factory.go:153] Registering CRI-O factory Jul 16 21:47:31.915315 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.734139 2585 factory.go:223] Registration of the crio container factory successfully Jul 16 21:47:31.915315 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.734179 2585 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 16 21:47:31.915315 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.734195 2585 factory.go:103] Registering Raw factory Jul 16 21:47:31.915315 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.734206 2585 manager.go:1196] Started watching for new ooms in manager Jul 16 21:47:31.915315 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.734662 2585 manager.go:349] Starting recovery of all containers Jul 16 21:47:31.915315 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.745747 2585 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 21:47:31.915315 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.746339 2585 manager.go:354] Recovery completed Jul 16 21:47:31.915315 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:31.749612 2585 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-128-255.ec2.internal\" not found" node="ip-10-0-128-255.ec2.internal" Jul 16 21:47:31.915315 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.750818 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 21:47:31.915315 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.752979 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-255.ec2.internal" event="NodeHasSufficientMemory" Jul 16 21:47:31.915315 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.753000 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-255.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 21:47:31.915315 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.753010 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-255.ec2.internal" event="NodeHasSufficientPID" Jul 16 21:47:31.915315 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.753545 2585 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 16 21:47:31.915315 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.753556 2585 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 16 21:47:31.915315 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.753563 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 21:47:31.915315 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.753587 2585 state_mem.go:36] "Initialized new in-memory state store" Jul 16 21:47:31.915315 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.754393 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-255.ec2.internal" event="NodeHasSufficientMemory" Jul 16 21:47:31.915315 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.754415 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-255.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 21:47:31.915315 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.754425 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-255.ec2.internal" event="NodeHasSufficientPID" Jul 16 21:47:31.915315 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.754802 2585 policy_none.go:49] "None policy: Start" Jul 16 21:47:31.915315 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.754815 2585 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 16 21:47:31.915315 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.754825 2585 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 16 21:47:31.915315 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.755726 2585 policy_none.go:47] "Start" Jul 16 21:47:31.915315 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.806519 2585 manager.go:337] "Starting Device Plugin manager" Jul 16 21:47:31.915315 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:31.806707 2585 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 16 21:47:31.915315 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.806720 2585 server.go:87] "Starting device plugin registration server" Jul 16 21:47:31.915315 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.806951 2585 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 16 21:47:31.915315 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.806962 2585 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 16 21:47:31.915315 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.807318 2585 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 16 21:47:31.915315 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.807386 2585 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 16 21:47:31.915315 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.807395 2585 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 16 21:47:31.915315 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:31.807999 2585 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 16 21:47:31.915315 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:31.808046 2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-255.ec2.internal\" not found" Jul 16 21:47:31.915315 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.907587 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 21:47:31.915315 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.909426 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-255.ec2.internal" event="NodeHasSufficientMemory" Jul 16 21:47:31.915315 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.909451 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-255.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 21:47:31.915315 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.909464 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-255.ec2.internal" event="NodeHasSufficientPID" Jul 16 21:47:31.915315 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.909490 2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-255.ec2.internal" Jul 16 21:47:31.916643 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.916391 2585 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-255.ec2.internal" Jul 16 21:47:31.916643 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:31.916416 2585 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-255.ec2.internal\": node \"ip-10-0-128-255.ec2.internal\" not found" Jul 16 21:47:32.009507 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.927013 2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 16 21:47:32.009507 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.928102 2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 16 21:47:32.009507 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.928137 2585 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 16 21:47:32.009507 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.928158 2585 kubelet.go:2448] "Starting kubelet main sync loop" Jul 16 21:47:32.009507 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:31.928188 2585 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 16 21:47:32.009507 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:31.930976 2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 21:47:32.009507 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:31.937812 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-255.ec2.internal\" not found" Jul 16 21:47:32.127544 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:32.029076 2585 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-255.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-255.ec2.internal"] Jul 16 21:47:32.127544 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:32.029154 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 21:47:32.127544 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:32.030675 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-255.ec2.internal" event="NodeHasSufficientMemory" Jul 16 21:47:32.127544 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:32.030701 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-255.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 21:47:32.127544 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:32.030711 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-255.ec2.internal" event="NodeHasSufficientPID" Jul 16 21:47:32.127544 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:32.032150 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 21:47:32.127544 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:32.032300 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-255.ec2.internal" Jul 16 21:47:32.127544 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:32.032331 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 21:47:32.127544 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:32.032822 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-255.ec2.internal" event="NodeHasSufficientMemory" Jul 16 21:47:32.127544 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:32.032829 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-255.ec2.internal" event="NodeHasSufficientMemory" Jul 16 21:47:32.127544 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:32.032851 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-255.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 21:47:32.127544 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:32.032851 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-255.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 21:47:32.127544 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:32.032862 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-255.ec2.internal" event="NodeHasSufficientPID" Jul 16 21:47:32.127544 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:32.032865 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-255.ec2.internal" event="NodeHasSufficientPID" Jul 16 21:47:32.127544 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:32.034363 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-255.ec2.internal" Jul 16 21:47:32.127544 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:32.034401 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 21:47:32.127544 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:32.035038 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-255.ec2.internal" event="NodeHasSufficientMemory" Jul 16 21:47:32.127544 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:32.035059 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-255.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 21:47:32.127544 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:32.035069 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-255.ec2.internal" event="NodeHasSufficientPID" Jul 16 21:47:32.127544 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:32.037859 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-255.ec2.internal\" not found" Jul 16 21:47:32.127544 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:32.069835 2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-255.ec2.internal\" not found" node="ip-10-0-128-255.ec2.internal" Jul 16 21:47:32.127544 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:32.074154 2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-255.ec2.internal\" not found" node="ip-10-0-128-255.ec2.internal" Jul 16 21:47:32.127544 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:32.085156 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/227134a40e81cd87d8542159249ce174-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-255.ec2.internal\" (UID: \"227134a40e81cd87d8542159249ce174\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-255.ec2.internal" Jul 16 21:47:32.127544 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:32.085177 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/227134a40e81cd87d8542159249ce174-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-255.ec2.internal\" (UID: \"227134a40e81cd87d8542159249ce174\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-255.ec2.internal" Jul 16 21:47:32.127544 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:32.085195 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9c641fdc20deb5fa481c19356c3eeb80-config\") pod \"kube-apiserver-proxy-ip-10-0-128-255.ec2.internal\" (UID: \"9c641fdc20deb5fa481c19356c3eeb80\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-255.ec2.internal" Jul 16 21:47:32.226022 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:32.138165 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-255.ec2.internal\" not found" Jul 16 21:47:32.226022 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:32.186290 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/227134a40e81cd87d8542159249ce174-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-255.ec2.internal\" (UID: \"227134a40e81cd87d8542159249ce174\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-255.ec2.internal" Jul 16 21:47:32.226022 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:32.186313 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/227134a40e81cd87d8542159249ce174-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-255.ec2.internal\" (UID: \"227134a40e81cd87d8542159249ce174\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-255.ec2.internal" Jul 16 21:47:32.226022 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:32.186329 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9c641fdc20deb5fa481c19356c3eeb80-config\") pod \"kube-apiserver-proxy-ip-10-0-128-255.ec2.internal\" (UID: \"9c641fdc20deb5fa481c19356c3eeb80\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-255.ec2.internal" Jul 16 21:47:32.226022 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:32.186366 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9c641fdc20deb5fa481c19356c3eeb80-config\") pod \"kube-apiserver-proxy-ip-10-0-128-255.ec2.internal\" (UID: \"9c641fdc20deb5fa481c19356c3eeb80\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-255.ec2.internal" Jul 16 21:47:32.226022 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:32.186393 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/227134a40e81cd87d8542159249ce174-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-255.ec2.internal\" (UID: \"227134a40e81cd87d8542159249ce174\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-255.ec2.internal" Jul 16 21:47:32.226022 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:32.186403 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/227134a40e81cd87d8542159249ce174-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-255.ec2.internal\" (UID: \"227134a40e81cd87d8542159249ce174\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-255.ec2.internal" Jul 16 21:47:32.314520 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:32.239003 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-255.ec2.internal\" not found" Jul 16 21:47:32.438999 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:32.339180 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-255.ec2.internal\" not found" Jul 16 21:47:32.438999 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:32.373964 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-255.ec2.internal" Jul 16 21:47:32.438999 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:32.377966 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-255.ec2.internal" Jul 16 21:47:32.440015 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:32.439877 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-255.ec2.internal\" not found" Jul 16 21:47:32.543111 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:32.542718 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-255.ec2.internal\" not found" Jul 16 21:47:32.736597 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:32.643209 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-255.ec2.internal\" not found" Jul 16 21:47:32.736597 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:32.670762 2585 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 16 21:47:32.736597 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:32.670949 2585 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 16 21:47:32.736597 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:32.670993 2585 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 16 21:47:32.736597 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:32.686854 2585 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 21:47:32.862745 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:32.748166 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-255.ec2.internal\" not found" Jul 16 21:47:32.862745 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:32.748215 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 21:42:31 +0000 UTC" deadline="2028-02-14 19:52:49.547443381 +0000 UTC" Jul 16 21:47:32.862745 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:32.748258 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13870h5m16.799189093s" Jul 16 21:47:32.862745 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:32.848730 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-255.ec2.internal\" not found" Jul 16 21:47:32.983898 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:32.949208 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-255.ec2.internal\" not found" Jul 16 21:47:33.101065 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:33.050018 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-255.ec2.internal\" not found" Jul 16 21:47:33.101065 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:33.094613 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c641fdc20deb5fa481c19356c3eeb80.slice/crio-a055ebed9a8c1981631413a499b5cf3287fd48aaa9964513b2523daa46980a3b WatchSource:0}: Error finding container a055ebed9a8c1981631413a499b5cf3287fd48aaa9964513b2523daa46980a3b: Status 404 returned error can't find the container with id a055ebed9a8c1981631413a499b5cf3287fd48aaa9964513b2523daa46980a3b Jul 16 21:47:33.101065 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:33.094940 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod227134a40e81cd87d8542159249ce174.slice/crio-edec36e3e6058d04097cdb16a3a2c00303470206d91f92e3af1ed57b348eb6a9 WatchSource:0}: Error finding container edec36e3e6058d04097cdb16a3a2c00303470206d91f92e3af1ed57b348eb6a9: Status 404 returned error can't find the container with id edec36e3e6058d04097cdb16a3a2c00303470206d91f92e3af1ed57b348eb6a9 Jul 16 21:47:33.101065 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.099373 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 21:47:33.211063 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:33.150671 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-255.ec2.internal\" not found" Jul 16 21:47:33.251612 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:33.251575 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-255.ec2.internal\" not found" Jul 16 21:47:33.288841 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.288814 2585 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 21:47:33.335849 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.335800 2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-255.ec2.internal" Jul 16 21:47:33.350955 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.350934 2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 21:47:33.351943 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.351930 2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-255.ec2.internal" Jul 16 21:47:33.364588 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.364565 2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 21:47:33.593973 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.593949 2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 21:47:33.714706 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.714666 2585 apiserver.go:52] "Watching apiserver" Jul 16 21:47:33.724177 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.724149 2585 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 16 21:47:33.724701 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.724674 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-255.ec2.internal","openshift-multus/multus-additional-cni-plugins-998qs","kube-system/kube-apiserver-proxy-ip-10-0-128-255.ec2.internal","openshift-multus/multus-ntnh7","openshift-multus/network-metrics-daemon-lcl48","openshift-network-diagnostics/network-check-target-zvq8m","openshift-network-operator/iptables-alerter-r4g7k","openshift-ovn-kubernetes/ovnkube-node-lmrcx","kube-system/global-pull-secret-syncer-jwlrc","kube-system/konnectivity-agent-lfwng","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djdlj","openshift-cluster-node-tuning-operator/tuned-lbh7v","openshift-dns/node-resolver-7tltv","openshift-image-registry/node-ca-65cgd"] Jul 16 21:47:33.727480 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.727418 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r4g7k" Jul 16 21:47:33.730024 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.729992 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 16 21:47:33.730148 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.730002 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 16 21:47:33.730148 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.730054 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 16 21:47:33.730432 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.730408 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-47nfj\"" Jul 16 21:47:33.730818 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.730735 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7tltv" Jul 16 21:47:33.733324 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.733288 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 16 21:47:33.733623 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.733551 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cvhpb\"" Jul 16 21:47:33.733974 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.733840 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 16 21:47:33.734116 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.734099 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-65cgd" Jul 16 21:47:33.736433 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.736194 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 16 21:47:33.736433 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.736226 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qj4t8\"" Jul 16 21:47:33.736595 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.736577 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 16 21:47:33.737046 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.736881 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 16 21:47:33.737046 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.737034 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ntnh7" Jul 16 21:47:33.739370 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.739353 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcl48" Jul 16 21:47:33.739442 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:33.739429 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcl48" podUID="c33b5633-01c6-4f48-8cb7-6969a24fe86b" Jul 16 21:47:33.740526 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.740499 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 16 21:47:33.740605 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.740550 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 16 21:47:33.740674 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.740610 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 16 21:47:33.740719 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.740709 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 16 21:47:33.741169 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.740910 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mxf44\"" Jul 16 21:47:33.741986 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.741962 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvq8m" Jul 16 21:47:33.742079 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:33.742040 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvq8m" podUID="f79c076a-55a4-487c-a763-9a9923ba4129" Jul 16 21:47:33.744704 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.744617 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwlrc" Jul 16 21:47:33.744704 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:33.744674 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwlrc" podUID="e9b9235a-c4fc-4aac-93b4-2964356667f1" Jul 16 21:47:33.747036 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.747014 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:33.749302 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.749282 2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 16 21:47:33.750360 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.750135 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-998qs" Jul 16 21:47:33.752595 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.752573 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-df6cq\"" Jul 16 21:47:33.753078 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.753053 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 16 21:47:33.756097 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.753486 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 16 21:47:33.756097 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.754033 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 16 21:47:33.756097 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.754259 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 16 21:47:33.756097 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.754547 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 16 21:47:33.756097 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.755233 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 16 21:47:33.756097 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.755791 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lfwng" Jul 16 21:47:33.758129 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.757637 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 16 21:47:33.758129 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.757871 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jlqm2\"" Jul 16 21:47:33.758129 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.757904 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 16 21:47:33.760396 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.760369 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djdlj" Jul 16 21:47:33.761561 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.761131 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 16 21:47:33.761561 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.761175 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-k5bvz\"" Jul 16 21:47:33.761561 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.761410 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 16 21:47:33.763031 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.763010 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" Jul 16 21:47:33.763829 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.763670 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 16 21:47:33.764025 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.764006 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 16 21:47:33.765251 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.764452 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 16 21:47:33.765251 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.764656 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5dk5s\"" Jul 16 21:47:33.765251 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.764997 2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 21:47:33.765469 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.765328 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 16 21:47:33.765917 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.765758 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 16 21:47:33.766260 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.766127 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6bzxw\"" Jul 16 21:47:33.794387 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.794358 2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-pgzgw" Jul 16 21:47:33.796765 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.796742 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwk8v\" (UniqueName: \"kubernetes.io/projected/f64092be-22df-4e1c-ab7e-4fcc7afb7ec7-kube-api-access-xwk8v\") pod \"node-ca-65cgd\" (UID: \"f64092be-22df-4e1c-ab7e-4fcc7afb7ec7\") " pod="openshift-image-registry/node-ca-65cgd" Jul 16 21:47:33.796874 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.796787 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9a9ccc80-b805-462b-87b9-22cfdc7e073c-iptables-alerter-script\") pod \"iptables-alerter-r4g7k\" (UID: \"9a9ccc80-b805-462b-87b9-22cfdc7e073c\") " pod="openshift-network-operator/iptables-alerter-r4g7k" Jul 16 21:47:33.796874 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.796815 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1dabb80a-8529-4289-82e4-e44ca61494c8-hosts-file\") pod \"node-resolver-7tltv\" (UID: \"1dabb80a-8529-4289-82e4-e44ca61494c8\") " pod="openshift-dns/node-resolver-7tltv" Jul 16 21:47:33.797005 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.796893 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dabb80a-8529-4289-82e4-e44ca61494c8-tmp-dir\") pod \"node-resolver-7tltv\" (UID: \"1dabb80a-8529-4289-82e4-e44ca61494c8\") " pod="openshift-dns/node-resolver-7tltv" Jul 16 21:47:33.797005 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.796934 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9a9ccc80-b805-462b-87b9-22cfdc7e073c-host-slash\") pod \"iptables-alerter-r4g7k\" (UID: \"9a9ccc80-b805-462b-87b9-22cfdc7e073c\") " pod="openshift-network-operator/iptables-alerter-r4g7k" Jul 16 21:47:33.797005 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.796965 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xk47\" (UniqueName: \"kubernetes.io/projected/9a9ccc80-b805-462b-87b9-22cfdc7e073c-kube-api-access-5xk47\") pod \"iptables-alerter-r4g7k\" (UID: \"9a9ccc80-b805-462b-87b9-22cfdc7e073c\") " pod="openshift-network-operator/iptables-alerter-r4g7k" Jul 16 21:47:33.797005 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.796989 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njtb2\" (UniqueName: \"kubernetes.io/projected/1dabb80a-8529-4289-82e4-e44ca61494c8-kube-api-access-njtb2\") pod \"node-resolver-7tltv\" (UID: \"1dabb80a-8529-4289-82e4-e44ca61494c8\") " pod="openshift-dns/node-resolver-7tltv" Jul 16 21:47:33.797005 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.797005 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f64092be-22df-4e1c-ab7e-4fcc7afb7ec7-host\") pod \"node-ca-65cgd\" (UID: \"f64092be-22df-4e1c-ab7e-4fcc7afb7ec7\") " pod="openshift-image-registry/node-ca-65cgd" Jul 16 21:47:33.797266 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.797020 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f64092be-22df-4e1c-ab7e-4fcc7afb7ec7-serviceca\") pod \"node-ca-65cgd\" (UID: \"f64092be-22df-4e1c-ab7e-4fcc7afb7ec7\") " pod="openshift-image-registry/node-ca-65cgd" Jul 16 21:47:33.803898 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.803875 2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-pgzgw" Jul 16 21:47:33.834942 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.834853 2585 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 16 21:47:33.897536 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.897463 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9b9235a-c4fc-4aac-93b4-2964356667f1-original-pull-secret\") pod \"global-pull-secret-syncer-jwlrc\" (UID: \"e9b9235a-c4fc-4aac-93b4-2964356667f1\") " pod="kube-system/global-pull-secret-syncer-jwlrc" Jul 16 21:47:33.897536 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.897496 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6e7f0775-368f-4f7b-be25-3bf85b42f973-node-log\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:33.897536 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.897515 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0b72c656-880f-40d7-936f-4f46dc9d9128-registration-dir\") pod \"aws-ebs-csi-driver-node-djdlj\" (UID: \"0b72c656-880f-40d7-936f-4f46dc9d9128\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djdlj" Jul 16 21:47:33.897536 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.897539 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3ad1637f-5724-4467-a756-2b059ff5e457-multus-socket-dir-parent\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:33.898501 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.897562 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww8b5\" (UniqueName: \"kubernetes.io/projected/3ad1637f-5724-4467-a756-2b059ff5e457-kube-api-access-ww8b5\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:33.898501 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.897592 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0b72c656-880f-40d7-936f-4f46dc9d9128-socket-dir\") pod \"aws-ebs-csi-driver-node-djdlj\" (UID: \"0b72c656-880f-40d7-936f-4f46dc9d9128\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djdlj" Jul 16 21:47:33.898501 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.897607 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnfhp\" (UniqueName: \"kubernetes.io/projected/0b72c656-880f-40d7-936f-4f46dc9d9128-kube-api-access-wnfhp\") pod \"aws-ebs-csi-driver-node-djdlj\" (UID: \"0b72c656-880f-40d7-936f-4f46dc9d9128\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djdlj" Jul 16 21:47:33.898501 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.897628 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1dabb80a-8529-4289-82e4-e44ca61494c8-hosts-file\") pod \"node-resolver-7tltv\" (UID: \"1dabb80a-8529-4289-82e4-e44ca61494c8\") " pod="openshift-dns/node-resolver-7tltv" Jul 16 21:47:33.898501 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.897649 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dabb80a-8529-4289-82e4-e44ca61494c8-tmp-dir\") pod \"node-resolver-7tltv\" (UID: \"1dabb80a-8529-4289-82e4-e44ca61494c8\") " pod="openshift-dns/node-resolver-7tltv" Jul 16 21:47:33.898501 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.897665 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tbd8\" (UniqueName: \"kubernetes.io/projected/58e464a1-1e18-4f00-805d-46c3f88ebd51-kube-api-access-8tbd8\") pod \"tuned-lbh7v\" (UID: \"58e464a1-1e18-4f00-805d-46c3f88ebd51\") " pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" Jul 16 21:47:33.898501 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.897686 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e7f0775-368f-4f7b-be25-3bf85b42f973-var-lib-openvswitch\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:33.898501 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.897712 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0b72c656-880f-40d7-936f-4f46dc9d9128-device-dir\") pod \"aws-ebs-csi-driver-node-djdlj\" (UID: \"0b72c656-880f-40d7-936f-4f46dc9d9128\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djdlj" Jul 16 21:47:33.898501 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.897728 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/58e464a1-1e18-4f00-805d-46c3f88ebd51-etc-sysctl-d\") pod \"tuned-lbh7v\" (UID: \"58e464a1-1e18-4f00-805d-46c3f88ebd51\") " pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" Jul 16 21:47:33.898501 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.897743 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/58e464a1-1e18-4f00-805d-46c3f88ebd51-lib-modules\") pod \"tuned-lbh7v\" (UID: \"58e464a1-1e18-4f00-805d-46c3f88ebd51\") " pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" Jul 16 21:47:33.898501 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.897757 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6e7f0775-368f-4f7b-be25-3bf85b42f973-host-slash\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:33.898501 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.897775 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e7f0775-368f-4f7b-be25-3bf85b42f973-host-run-ovn-kubernetes\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:33.898501 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.897792 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6e7f0775-368f-4f7b-be25-3bf85b42f973-ovn-node-metrics-cert\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:33.898501 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.897820 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3ad1637f-5724-4467-a756-2b059ff5e457-host-var-lib-cni-bin\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:33.898501 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.897861 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ad1637f-5724-4467-a756-2b059ff5e457-host-var-lib-kubelet\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:33.898501 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.897885 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6e7f0775-368f-4f7b-be25-3bf85b42f973-systemd-units\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:33.899379 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.897945 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6e7f0775-368f-4f7b-be25-3bf85b42f973-run-ovn\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:33.899379 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.897969 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6e7f0775-368f-4f7b-be25-3bf85b42f973-log-socket\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:33.899379 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.897993 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0b72c656-880f-40d7-936f-4f46dc9d9128-kubelet-dir\") pod \"aws-ebs-csi-driver-node-djdlj\" (UID: \"0b72c656-880f-40d7-936f-4f46dc9d9128\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djdlj" Jul 16 21:47:33.899379 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.898017 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3ad1637f-5724-4467-a756-2b059ff5e457-host-run-multus-certs\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:33.899379 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.898040 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/58e464a1-1e18-4f00-805d-46c3f88ebd51-etc-systemd\") pod \"tuned-lbh7v\" (UID: \"58e464a1-1e18-4f00-805d-46c3f88ebd51\") " pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" Jul 16 21:47:33.899379 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.898067 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58e464a1-1e18-4f00-805d-46c3f88ebd51-host\") pod \"tuned-lbh7v\" (UID: \"58e464a1-1e18-4f00-805d-46c3f88ebd51\") " pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" Jul 16 21:47:33.899379 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.898090 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58e464a1-1e18-4f00-805d-46c3f88ebd51-tmp\") pod \"tuned-lbh7v\" (UID: \"58e464a1-1e18-4f00-805d-46c3f88ebd51\") " pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" Jul 16 21:47:33.899379 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.898107 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf8tf\" (UniqueName: \"kubernetes.io/projected/c33b5633-01c6-4f48-8cb7-6969a24fe86b-kube-api-access-vf8tf\") pod \"network-metrics-daemon-lcl48\" (UID: \"c33b5633-01c6-4f48-8cb7-6969a24fe86b\") " pod="openshift-multus/network-metrics-daemon-lcl48" Jul 16 21:47:33.899379 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.898130 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6e7f0775-368f-4f7b-be25-3bf85b42f973-ovnkube-config\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:33.899379 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.898151 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ad1637f-5724-4467-a756-2b059ff5e457-cnibin\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:33.899379 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.898165 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3ad1637f-5724-4467-a756-2b059ff5e457-os-release\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:33.899379 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.898196 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3ad1637f-5724-4467-a756-2b059ff5e457-cni-binary-copy\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:33.899379 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.898219 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3ad1637f-5724-4467-a756-2b059ff5e457-multus-daemon-config\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:33.899379 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.898248 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e7f0775-368f-4f7b-be25-3bf85b42f973-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:33.899379 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.898287 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a53c2dc1-5fa4-493e-b251-7438ccefcc23-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-998qs\" (UID: \"a53c2dc1-5fa4-493e-b251-7438ccefcc23\") " pod="openshift-multus/multus-additional-cni-plugins-998qs" Jul 16 21:47:33.899379 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.898328 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3ad1637f-5724-4467-a756-2b059ff5e457-host-run-netns\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:33.899379 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.898377 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ad1637f-5724-4467-a756-2b059ff5e457-etc-kubernetes\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:33.900272 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.898396 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9a9ccc80-b805-462b-87b9-22cfdc7e073c-iptables-alerter-script\") pod \"iptables-alerter-r4g7k\" (UID: \"9a9ccc80-b805-462b-87b9-22cfdc7e073c\") " pod="openshift-network-operator/iptables-alerter-r4g7k" Jul 16 21:47:33.900272 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.898412 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a53c2dc1-5fa4-493e-b251-7438ccefcc23-os-release\") pod \"multus-additional-cni-plugins-998qs\" (UID: \"a53c2dc1-5fa4-493e-b251-7438ccefcc23\") " pod="openshift-multus/multus-additional-cni-plugins-998qs" Jul 16 21:47:33.900272 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.898428 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/58e464a1-1e18-4f00-805d-46c3f88ebd51-etc-sysconfig\") pod \"tuned-lbh7v\" (UID: \"58e464a1-1e18-4f00-805d-46c3f88ebd51\") " pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" Jul 16 21:47:33.900272 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.898449 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/43f80a88-5e25-4f60-980a-8bb22936ba6c-agent-certs\") pod \"konnectivity-agent-lfwng\" (UID: \"43f80a88-5e25-4f60-980a-8bb22936ba6c\") " pod="kube-system/konnectivity-agent-lfwng" Jul 16 21:47:33.900272 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.898465 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4hrg\" (UniqueName: \"kubernetes.io/projected/f79c076a-55a4-487c-a763-9a9923ba4129-kube-api-access-x4hrg\") pod \"network-check-target-zvq8m\" (UID: \"f79c076a-55a4-487c-a763-9a9923ba4129\") " pod="openshift-network-diagnostics/network-check-target-zvq8m" Jul 16 21:47:33.900272 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.898487 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f64092be-22df-4e1c-ab7e-4fcc7afb7ec7-serviceca\") pod \"node-ca-65cgd\" (UID: \"f64092be-22df-4e1c-ab7e-4fcc7afb7ec7\") " pod="openshift-image-registry/node-ca-65cgd" Jul 16 21:47:33.900272 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.898516 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a53c2dc1-5fa4-493e-b251-7438ccefcc23-tuning-conf-dir\") pod \"multus-additional-cni-plugins-998qs\" (UID: \"a53c2dc1-5fa4-493e-b251-7438ccefcc23\") " pod="openshift-multus/multus-additional-cni-plugins-998qs" Jul 16 21:47:33.900272 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.898541 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58e464a1-1e18-4f00-805d-46c3f88ebd51-sys\") pod \"tuned-lbh7v\" (UID: \"58e464a1-1e18-4f00-805d-46c3f88ebd51\") " pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" Jul 16 21:47:33.900272 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.898584 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e7f0775-368f-4f7b-be25-3bf85b42f973-etc-openvswitch\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:33.900272 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.898609 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6e7f0775-368f-4f7b-be25-3bf85b42f973-host-cni-bin\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:33.900272 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.898638 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3ad1637f-5724-4467-a756-2b059ff5e457-hostroot\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:33.900272 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.898661 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/58e464a1-1e18-4f00-805d-46c3f88ebd51-etc-sysctl-conf\") pod \"tuned-lbh7v\" (UID: \"58e464a1-1e18-4f00-805d-46c3f88ebd51\") " pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" Jul 16 21:47:33.900272 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.898685 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58e464a1-1e18-4f00-805d-46c3f88ebd51-var-lib-kubelet\") pod \"tuned-lbh7v\" (UID: \"58e464a1-1e18-4f00-805d-46c3f88ebd51\") " pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" Jul 16 21:47:33.900272 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.898719 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/58e464a1-1e18-4f00-805d-46c3f88ebd51-etc-tuned\") pod \"tuned-lbh7v\" (UID: \"58e464a1-1e18-4f00-805d-46c3f88ebd51\") " pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" Jul 16 21:47:33.900272 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.898734 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3ad1637f-5724-4467-a756-2b059ff5e457-host-run-k8s-cni-cncf-io\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:33.900272 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.898772 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58e464a1-1e18-4f00-805d-46c3f88ebd51-etc-kubernetes\") pod \"tuned-lbh7v\" (UID: \"58e464a1-1e18-4f00-805d-46c3f88ebd51\") " pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" Jul 16 21:47:33.901178 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.898789 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e9b9235a-c4fc-4aac-93b4-2964356667f1-dbus\") pod \"global-pull-secret-syncer-jwlrc\" (UID: \"e9b9235a-c4fc-4aac-93b4-2964356667f1\") " pod="kube-system/global-pull-secret-syncer-jwlrc" Jul 16 21:47:33.901178 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.898825 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6e7f0775-368f-4f7b-be25-3bf85b42f973-ovnkube-script-lib\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:33.901178 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.898840 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/43f80a88-5e25-4f60-980a-8bb22936ba6c-konnectivity-ca\") pod \"konnectivity-agent-lfwng\" (UID: \"43f80a88-5e25-4f60-980a-8bb22936ba6c\") " pod="kube-system/konnectivity-agent-lfwng" Jul 16 21:47:33.901178 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.898855 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0b72c656-880f-40d7-936f-4f46dc9d9128-sys-fs\") pod \"aws-ebs-csi-driver-node-djdlj\" (UID: \"0b72c656-880f-40d7-936f-4f46dc9d9128\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djdlj" Jul 16 21:47:33.901178 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.898888 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ad1637f-5724-4467-a756-2b059ff5e457-multus-cni-dir\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:33.901178 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.898908 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a53c2dc1-5fa4-493e-b251-7438ccefcc23-cni-binary-copy\") pod \"multus-additional-cni-plugins-998qs\" (UID: \"a53c2dc1-5fa4-493e-b251-7438ccefcc23\") " pod="openshift-multus/multus-additional-cni-plugins-998qs" Jul 16 21:47:33.901178 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.898923 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e7f0775-368f-4f7b-be25-3bf85b42f973-host-kubelet\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:33.901178 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.898938 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6e7f0775-368f-4f7b-be25-3bf85b42f973-host-cni-netd\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:33.901178 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.898956 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3ad1637f-5724-4467-a756-2b059ff5e457-host-var-lib-cni-multus\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:33.901178 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.898974 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ad1637f-5724-4467-a756-2b059ff5e457-multus-conf-dir\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:33.901178 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.898991 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f64092be-22df-4e1c-ab7e-4fcc7afb7ec7-host\") pod \"node-ca-65cgd\" (UID: \"f64092be-22df-4e1c-ab7e-4fcc7afb7ec7\") " pod="openshift-image-registry/node-ca-65cgd" Jul 16 21:47:33.901178 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.899005 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/58e464a1-1e18-4f00-805d-46c3f88ebd51-run\") pod \"tuned-lbh7v\" (UID: \"58e464a1-1e18-4f00-805d-46c3f88ebd51\") " pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" Jul 16 21:47:33.901178 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.899034 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6e7f0775-368f-4f7b-be25-3bf85b42f973-host-run-netns\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:33.901178 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.899052 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5hs4\" (UniqueName: \"kubernetes.io/projected/6e7f0775-368f-4f7b-be25-3bf85b42f973-kube-api-access-j5hs4\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:33.901178 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.899085 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0b72c656-880f-40d7-936f-4f46dc9d9128-etc-selinux\") pod \"aws-ebs-csi-driver-node-djdlj\" (UID: \"0b72c656-880f-40d7-936f-4f46dc9d9128\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djdlj" Jul 16 21:47:33.901178 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.899140 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a53c2dc1-5fa4-493e-b251-7438ccefcc23-cnibin\") pod \"multus-additional-cni-plugins-998qs\" (UID: \"a53c2dc1-5fa4-493e-b251-7438ccefcc23\") " pod="openshift-multus/multus-additional-cni-plugins-998qs" Jul 16 21:47:33.901997 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.899159 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c33b5633-01c6-4f48-8cb7-6969a24fe86b-metrics-certs\") pod \"network-metrics-daemon-lcl48\" (UID: \"c33b5633-01c6-4f48-8cb7-6969a24fe86b\") " pod="openshift-multus/network-metrics-daemon-lcl48" Jul 16 21:47:33.901997 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.899173 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e9b9235a-c4fc-4aac-93b4-2964356667f1-kubelet-config\") pod \"global-pull-secret-syncer-jwlrc\" (UID: \"e9b9235a-c4fc-4aac-93b4-2964356667f1\") " pod="kube-system/global-pull-secret-syncer-jwlrc" Jul 16 21:47:33.901997 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.899193 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6e7f0775-368f-4f7b-be25-3bf85b42f973-env-overrides\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:33.901997 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.899217 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ad1637f-5724-4467-a756-2b059ff5e457-system-cni-dir\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:33.901997 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.899238 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9a9ccc80-b805-462b-87b9-22cfdc7e073c-host-slash\") pod \"iptables-alerter-r4g7k\" (UID: \"9a9ccc80-b805-462b-87b9-22cfdc7e073c\") " pod="openshift-network-operator/iptables-alerter-r4g7k" Jul 16 21:47:33.901997 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.899261 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xk47\" (UniqueName: \"kubernetes.io/projected/9a9ccc80-b805-462b-87b9-22cfdc7e073c-kube-api-access-5xk47\") pod \"iptables-alerter-r4g7k\" (UID: \"9a9ccc80-b805-462b-87b9-22cfdc7e073c\") " pod="openshift-network-operator/iptables-alerter-r4g7k" Jul 16 21:47:33.901997 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.899277 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njtb2\" (UniqueName: \"kubernetes.io/projected/1dabb80a-8529-4289-82e4-e44ca61494c8-kube-api-access-njtb2\") pod \"node-resolver-7tltv\" (UID: \"1dabb80a-8529-4289-82e4-e44ca61494c8\") " pod="openshift-dns/node-resolver-7tltv" Jul 16 21:47:33.901997 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.899542 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1dabb80a-8529-4289-82e4-e44ca61494c8-hosts-file\") pod \"node-resolver-7tltv\" (UID: \"1dabb80a-8529-4289-82e4-e44ca61494c8\") " pod="openshift-dns/node-resolver-7tltv" Jul 16 21:47:33.901997 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.899311 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwk8v\" (UniqueName: \"kubernetes.io/projected/f64092be-22df-4e1c-ab7e-4fcc7afb7ec7-kube-api-access-xwk8v\") pod \"node-ca-65cgd\" (UID: \"f64092be-22df-4e1c-ab7e-4fcc7afb7ec7\") " pod="openshift-image-registry/node-ca-65cgd" Jul 16 21:47:33.901997 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.899893 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a53c2dc1-5fa4-493e-b251-7438ccefcc23-system-cni-dir\") pod \"multus-additional-cni-plugins-998qs\" (UID: \"a53c2dc1-5fa4-493e-b251-7438ccefcc23\") " pod="openshift-multus/multus-additional-cni-plugins-998qs" Jul 16 21:47:33.901997 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.899945 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/58e464a1-1e18-4f00-805d-46c3f88ebd51-etc-modprobe-d\") pod \"tuned-lbh7v\" (UID: \"58e464a1-1e18-4f00-805d-46c3f88ebd51\") " pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" Jul 16 21:47:33.901997 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.899988 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6e7f0775-368f-4f7b-be25-3bf85b42f973-run-systemd\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:33.901997 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.899834 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dabb80a-8529-4289-82e4-e44ca61494c8-tmp-dir\") pod \"node-resolver-7tltv\" (UID: \"1dabb80a-8529-4289-82e4-e44ca61494c8\") " pod="openshift-dns/node-resolver-7tltv" Jul 16 21:47:33.901997 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.900759 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9a9ccc80-b805-462b-87b9-22cfdc7e073c-iptables-alerter-script\") pod \"iptables-alerter-r4g7k\" (UID: \"9a9ccc80-b805-462b-87b9-22cfdc7e073c\") " pod="openshift-network-operator/iptables-alerter-r4g7k" Jul 16 21:47:33.901997 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.900866 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f64092be-22df-4e1c-ab7e-4fcc7afb7ec7-host\") pod \"node-ca-65cgd\" (UID: \"f64092be-22df-4e1c-ab7e-4fcc7afb7ec7\") " pod="openshift-image-registry/node-ca-65cgd" Jul 16 21:47:33.901997 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.901274 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9a9ccc80-b805-462b-87b9-22cfdc7e073c-host-slash\") pod \"iptables-alerter-r4g7k\" (UID: \"9a9ccc80-b805-462b-87b9-22cfdc7e073c\") " pod="openshift-network-operator/iptables-alerter-r4g7k" Jul 16 21:47:33.901997 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.901338 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e7f0775-368f-4f7b-be25-3bf85b42f973-run-openvswitch\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:33.902960 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.901380 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a53c2dc1-5fa4-493e-b251-7438ccefcc23-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-998qs\" (UID: \"a53c2dc1-5fa4-493e-b251-7438ccefcc23\") " pod="openshift-multus/multus-additional-cni-plugins-998qs" Jul 16 21:47:33.902960 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.901411 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f64092be-22df-4e1c-ab7e-4fcc7afb7ec7-serviceca\") pod \"node-ca-65cgd\" (UID: \"f64092be-22df-4e1c-ab7e-4fcc7afb7ec7\") " pod="openshift-image-registry/node-ca-65cgd" Jul 16 21:47:33.902960 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.901435 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbr9m\" (UniqueName: \"kubernetes.io/projected/a53c2dc1-5fa4-493e-b251-7438ccefcc23-kube-api-access-qbr9m\") pod \"multus-additional-cni-plugins-998qs\" (UID: \"a53c2dc1-5fa4-493e-b251-7438ccefcc23\") " pod="openshift-multus/multus-additional-cni-plugins-998qs" Jul 16 21:47:33.914405 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.914088 2585 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 16 21:47:33.918736 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.918711 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwk8v\" (UniqueName: \"kubernetes.io/projected/f64092be-22df-4e1c-ab7e-4fcc7afb7ec7-kube-api-access-xwk8v\") pod \"node-ca-65cgd\" (UID: \"f64092be-22df-4e1c-ab7e-4fcc7afb7ec7\") " pod="openshift-image-registry/node-ca-65cgd" Jul 16 21:47:33.918736 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.918722 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xk47\" (UniqueName: \"kubernetes.io/projected/9a9ccc80-b805-462b-87b9-22cfdc7e073c-kube-api-access-5xk47\") pod \"iptables-alerter-r4g7k\" (UID: \"9a9ccc80-b805-462b-87b9-22cfdc7e073c\") " pod="openshift-network-operator/iptables-alerter-r4g7k" Jul 16 21:47:33.920162 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.920141 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njtb2\" (UniqueName: \"kubernetes.io/projected/1dabb80a-8529-4289-82e4-e44ca61494c8-kube-api-access-njtb2\") pod \"node-resolver-7tltv\" (UID: \"1dabb80a-8529-4289-82e4-e44ca61494c8\") " pod="openshift-dns/node-resolver-7tltv" Jul 16 21:47:33.938367 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.938308 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-255.ec2.internal" event={"ID":"9c641fdc20deb5fa481c19356c3eeb80","Type":"ContainerStarted","Data":"a055ebed9a8c1981631413a499b5cf3287fd48aaa9964513b2523daa46980a3b"} Jul 16 21:47:33.938607 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.938579 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-255.ec2.internal" event={"ID":"227134a40e81cd87d8542159249ce174","Type":"ContainerStarted","Data":"edec36e3e6058d04097cdb16a3a2c00303470206d91f92e3af1ed57b348eb6a9"} Jul 16 21:47:33.964735 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:33.964699 2585 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 21:47:34.001880 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.001844 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/43f80a88-5e25-4f60-980a-8bb22936ba6c-konnectivity-ca\") pod \"konnectivity-agent-lfwng\" (UID: \"43f80a88-5e25-4f60-980a-8bb22936ba6c\") " pod="kube-system/konnectivity-agent-lfwng" Jul 16 21:47:34.001880 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.001882 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0b72c656-880f-40d7-936f-4f46dc9d9128-sys-fs\") pod \"aws-ebs-csi-driver-node-djdlj\" (UID: \"0b72c656-880f-40d7-936f-4f46dc9d9128\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djdlj" Jul 16 21:47:34.002057 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.001907 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ad1637f-5724-4467-a756-2b059ff5e457-multus-cni-dir\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:34.002057 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.001932 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a53c2dc1-5fa4-493e-b251-7438ccefcc23-cni-binary-copy\") pod \"multus-additional-cni-plugins-998qs\" (UID: \"a53c2dc1-5fa4-493e-b251-7438ccefcc23\") " pod="openshift-multus/multus-additional-cni-plugins-998qs" Jul 16 21:47:34.002057 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.001956 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e7f0775-368f-4f7b-be25-3bf85b42f973-host-kubelet\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:34.002057 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.001978 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6e7f0775-368f-4f7b-be25-3bf85b42f973-host-cni-netd\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:34.002057 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.002004 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3ad1637f-5724-4467-a756-2b059ff5e457-host-var-lib-cni-multus\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:34.002057 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.002026 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ad1637f-5724-4467-a756-2b059ff5e457-multus-conf-dir\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:34.002057 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.002049 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/58e464a1-1e18-4f00-805d-46c3f88ebd51-run\") pod \"tuned-lbh7v\" (UID: \"58e464a1-1e18-4f00-805d-46c3f88ebd51\") " pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" Jul 16 21:47:34.002458 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.002112 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e7f0775-368f-4f7b-be25-3bf85b42f973-host-kubelet\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:34.002458 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.002164 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6e7f0775-368f-4f7b-be25-3bf85b42f973-host-run-netns\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:34.002458 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.002194 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/58e464a1-1e18-4f00-805d-46c3f88ebd51-run\") pod \"tuned-lbh7v\" (UID: \"58e464a1-1e18-4f00-805d-46c3f88ebd51\") " pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" Jul 16 21:47:34.002458 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.002198 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5hs4\" (UniqueName: \"kubernetes.io/projected/6e7f0775-368f-4f7b-be25-3bf85b42f973-kube-api-access-j5hs4\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:34.002458 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.002225 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0b72c656-880f-40d7-936f-4f46dc9d9128-etc-selinux\") pod \"aws-ebs-csi-driver-node-djdlj\" (UID: \"0b72c656-880f-40d7-936f-4f46dc9d9128\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djdlj" Jul 16 21:47:34.002458 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.002269 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6e7f0775-368f-4f7b-be25-3bf85b42f973-host-cni-netd\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:34.002458 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.002336 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0b72c656-880f-40d7-936f-4f46dc9d9128-sys-fs\") pod \"aws-ebs-csi-driver-node-djdlj\" (UID: \"0b72c656-880f-40d7-936f-4f46dc9d9128\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djdlj" Jul 16 21:47:34.002458 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.002343 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3ad1637f-5724-4467-a756-2b059ff5e457-host-var-lib-cni-multus\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:34.002458 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.002379 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a53c2dc1-5fa4-493e-b251-7438ccefcc23-cnibin\") pod \"multus-additional-cni-plugins-998qs\" (UID: \"a53c2dc1-5fa4-493e-b251-7438ccefcc23\") " pod="openshift-multus/multus-additional-cni-plugins-998qs" Jul 16 21:47:34.002458 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.002390 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ad1637f-5724-4467-a756-2b059ff5e457-multus-conf-dir\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:34.002458 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.002271 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a53c2dc1-5fa4-493e-b251-7438ccefcc23-cnibin\") pod \"multus-additional-cni-plugins-998qs\" (UID: \"a53c2dc1-5fa4-493e-b251-7438ccefcc23\") " pod="openshift-multus/multus-additional-cni-plugins-998qs" Jul 16 21:47:34.002458 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.002437 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c33b5633-01c6-4f48-8cb7-6969a24fe86b-metrics-certs\") pod \"network-metrics-daemon-lcl48\" (UID: \"c33b5633-01c6-4f48-8cb7-6969a24fe86b\") " pod="openshift-multus/network-metrics-daemon-lcl48" Jul 16 21:47:34.003036 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.002481 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e9b9235a-c4fc-4aac-93b4-2964356667f1-kubelet-config\") pod \"global-pull-secret-syncer-jwlrc\" (UID: \"e9b9235a-c4fc-4aac-93b4-2964356667f1\") " pod="kube-system/global-pull-secret-syncer-jwlrc" Jul 16 21:47:34.003036 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.002508 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6e7f0775-368f-4f7b-be25-3bf85b42f973-env-overrides\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:34.003036 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.002524 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ad1637f-5724-4467-a756-2b059ff5e457-multus-cni-dir\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:34.003036 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.002533 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ad1637f-5724-4467-a756-2b059ff5e457-system-cni-dir\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:34.003036 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.002565 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a53c2dc1-5fa4-493e-b251-7438ccefcc23-system-cni-dir\") pod \"multus-additional-cni-plugins-998qs\" (UID: \"a53c2dc1-5fa4-493e-b251-7438ccefcc23\") " pod="openshift-multus/multus-additional-cni-plugins-998qs" Jul 16 21:47:34.003036 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.002569 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/43f80a88-5e25-4f60-980a-8bb22936ba6c-konnectivity-ca\") pod \"konnectivity-agent-lfwng\" (UID: \"43f80a88-5e25-4f60-980a-8bb22936ba6c\") " pod="kube-system/konnectivity-agent-lfwng" Jul 16 21:47:34.003036 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.002589 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/58e464a1-1e18-4f00-805d-46c3f88ebd51-etc-modprobe-d\") pod \"tuned-lbh7v\" (UID: \"58e464a1-1e18-4f00-805d-46c3f88ebd51\") " pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" Jul 16 21:47:34.003036 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.002613 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6e7f0775-368f-4f7b-be25-3bf85b42f973-run-systemd\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:34.003036 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.002641 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e7f0775-368f-4f7b-be25-3bf85b42f973-run-openvswitch\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:34.003036 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.002669 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a53c2dc1-5fa4-493e-b251-7438ccefcc23-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-998qs\" (UID: \"a53c2dc1-5fa4-493e-b251-7438ccefcc23\") " pod="openshift-multus/multus-additional-cni-plugins-998qs" Jul 16 21:47:34.003036 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.002699 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbr9m\" (UniqueName: \"kubernetes.io/projected/a53c2dc1-5fa4-493e-b251-7438ccefcc23-kube-api-access-qbr9m\") pod \"multus-additional-cni-plugins-998qs\" (UID: \"a53c2dc1-5fa4-493e-b251-7438ccefcc23\") " pod="openshift-multus/multus-additional-cni-plugins-998qs" Jul 16 21:47:34.003036 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.002726 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9b9235a-c4fc-4aac-93b4-2964356667f1-original-pull-secret\") pod \"global-pull-secret-syncer-jwlrc\" (UID: \"e9b9235a-c4fc-4aac-93b4-2964356667f1\") " pod="kube-system/global-pull-secret-syncer-jwlrc" Jul 16 21:47:34.003036 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.002756 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6e7f0775-368f-4f7b-be25-3bf85b42f973-node-log\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:34.003036 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.002781 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0b72c656-880f-40d7-936f-4f46dc9d9128-registration-dir\") pod \"aws-ebs-csi-driver-node-djdlj\" (UID: \"0b72c656-880f-40d7-936f-4f46dc9d9128\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djdlj" Jul 16 21:47:34.003036 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.002805 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3ad1637f-5724-4467-a756-2b059ff5e457-multus-socket-dir-parent\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:34.003036 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.002835 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww8b5\" (UniqueName: \"kubernetes.io/projected/3ad1637f-5724-4467-a756-2b059ff5e457-kube-api-access-ww8b5\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:34.003036 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.002859 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0b72c656-880f-40d7-936f-4f46dc9d9128-socket-dir\") pod \"aws-ebs-csi-driver-node-djdlj\" (UID: \"0b72c656-880f-40d7-936f-4f46dc9d9128\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djdlj" Jul 16 21:47:34.003896 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.002885 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnfhp\" (UniqueName: \"kubernetes.io/projected/0b72c656-880f-40d7-936f-4f46dc9d9128-kube-api-access-wnfhp\") pod \"aws-ebs-csi-driver-node-djdlj\" (UID: \"0b72c656-880f-40d7-936f-4f46dc9d9128\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djdlj" Jul 16 21:47:34.003896 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.002914 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tbd8\" (UniqueName: \"kubernetes.io/projected/58e464a1-1e18-4f00-805d-46c3f88ebd51-kube-api-access-8tbd8\") pod \"tuned-lbh7v\" (UID: \"58e464a1-1e18-4f00-805d-46c3f88ebd51\") " pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" Jul 16 21:47:34.003896 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.002938 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e7f0775-368f-4f7b-be25-3bf85b42f973-var-lib-openvswitch\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:34.003896 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.002975 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0b72c656-880f-40d7-936f-4f46dc9d9128-device-dir\") pod \"aws-ebs-csi-driver-node-djdlj\" (UID: \"0b72c656-880f-40d7-936f-4f46dc9d9128\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djdlj" Jul 16 21:47:34.003896 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.002999 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/58e464a1-1e18-4f00-805d-46c3f88ebd51-etc-sysctl-d\") pod \"tuned-lbh7v\" (UID: \"58e464a1-1e18-4f00-805d-46c3f88ebd51\") " pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" Jul 16 21:47:34.003896 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.003025 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/58e464a1-1e18-4f00-805d-46c3f88ebd51-lib-modules\") pod \"tuned-lbh7v\" (UID: \"58e464a1-1e18-4f00-805d-46c3f88ebd51\") " pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" Jul 16 21:47:34.003896 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.003053 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6e7f0775-368f-4f7b-be25-3bf85b42f973-host-slash\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:34.003896 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.003079 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e7f0775-368f-4f7b-be25-3bf85b42f973-host-run-ovn-kubernetes\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:34.003896 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.003105 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6e7f0775-368f-4f7b-be25-3bf85b42f973-ovn-node-metrics-cert\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:34.003896 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.003127 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a53c2dc1-5fa4-493e-b251-7438ccefcc23-cni-binary-copy\") pod \"multus-additional-cni-plugins-998qs\" (UID: \"a53c2dc1-5fa4-493e-b251-7438ccefcc23\") " pod="openshift-multus/multus-additional-cni-plugins-998qs" Jul 16 21:47:34.003896 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.003130 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3ad1637f-5724-4467-a756-2b059ff5e457-host-var-lib-cni-bin\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:34.003896 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.003174 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ad1637f-5724-4467-a756-2b059ff5e457-host-var-lib-kubelet\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:34.003896 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.003196 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6e7f0775-368f-4f7b-be25-3bf85b42f973-host-run-netns\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:34.003896 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.003239 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6e7f0775-368f-4f7b-be25-3bf85b42f973-systemd-units\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:34.003896 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.003265 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6e7f0775-368f-4f7b-be25-3bf85b42f973-run-ovn\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:34.003896 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.003310 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6e7f0775-368f-4f7b-be25-3bf85b42f973-log-socket\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:34.003896 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.003353 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0b72c656-880f-40d7-936f-4f46dc9d9128-kubelet-dir\") pod \"aws-ebs-csi-driver-node-djdlj\" (UID: \"0b72c656-880f-40d7-936f-4f46dc9d9128\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djdlj" Jul 16 21:47:34.004767 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.003381 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3ad1637f-5724-4467-a756-2b059ff5e457-host-run-multus-certs\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:34.004767 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.003408 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/58e464a1-1e18-4f00-805d-46c3f88ebd51-etc-systemd\") pod \"tuned-lbh7v\" (UID: \"58e464a1-1e18-4f00-805d-46c3f88ebd51\") " pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" Jul 16 21:47:34.004767 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.003402 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0b72c656-880f-40d7-936f-4f46dc9d9128-etc-selinux\") pod \"aws-ebs-csi-driver-node-djdlj\" (UID: \"0b72c656-880f-40d7-936f-4f46dc9d9128\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djdlj" Jul 16 21:47:34.004767 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.003436 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58e464a1-1e18-4f00-805d-46c3f88ebd51-host\") pod \"tuned-lbh7v\" (UID: \"58e464a1-1e18-4f00-805d-46c3f88ebd51\") " pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" Jul 16 21:47:34.004767 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.003424 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6e7f0775-368f-4f7b-be25-3bf85b42f973-host-slash\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:34.004767 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.003465 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58e464a1-1e18-4f00-805d-46c3f88ebd51-tmp\") pod \"tuned-lbh7v\" (UID: \"58e464a1-1e18-4f00-805d-46c3f88ebd51\") " pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" Jul 16 21:47:34.004767 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.003640 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e7f0775-368f-4f7b-be25-3bf85b42f973-host-run-ovn-kubernetes\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:34.004767 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.003674 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0b72c656-880f-40d7-936f-4f46dc9d9128-registration-dir\") pod \"aws-ebs-csi-driver-node-djdlj\" (UID: \"0b72c656-880f-40d7-936f-4f46dc9d9128\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djdlj" Jul 16 21:47:34.004767 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.003708 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6e7f0775-368f-4f7b-be25-3bf85b42f973-run-systemd\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:34.004767 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.003715 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0b72c656-880f-40d7-936f-4f46dc9d9128-socket-dir\") pod \"aws-ebs-csi-driver-node-djdlj\" (UID: \"0b72c656-880f-40d7-936f-4f46dc9d9128\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djdlj" Jul 16 21:47:34.004767 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.003766 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/58e464a1-1e18-4f00-805d-46c3f88ebd51-lib-modules\") pod \"tuned-lbh7v\" (UID: \"58e464a1-1e18-4f00-805d-46c3f88ebd51\") " pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" Jul 16 21:47:34.004767 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.003819 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e7f0775-368f-4f7b-be25-3bf85b42f973-var-lib-openvswitch\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:34.004767 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.004056 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/58e464a1-1e18-4f00-805d-46c3f88ebd51-etc-sysctl-d\") pod \"tuned-lbh7v\" (UID: \"58e464a1-1e18-4f00-805d-46c3f88ebd51\") " pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" Jul 16 21:47:34.004767 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.004062 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0b72c656-880f-40d7-936f-4f46dc9d9128-device-dir\") pod \"aws-ebs-csi-driver-node-djdlj\" (UID: \"0b72c656-880f-40d7-936f-4f46dc9d9128\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djdlj" Jul 16 21:47:34.004767 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.004095 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf8tf\" (UniqueName: \"kubernetes.io/projected/c33b5633-01c6-4f48-8cb7-6969a24fe86b-kube-api-access-vf8tf\") pod \"network-metrics-daemon-lcl48\" (UID: \"c33b5633-01c6-4f48-8cb7-6969a24fe86b\") " pod="openshift-multus/network-metrics-daemon-lcl48" Jul 16 21:47:34.004767 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.004125 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6e7f0775-368f-4f7b-be25-3bf85b42f973-ovnkube-config\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:34.004767 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.004152 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ad1637f-5724-4467-a756-2b059ff5e457-cnibin\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:34.005599 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.004168 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a53c2dc1-5fa4-493e-b251-7438ccefcc23-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-998qs\" (UID: \"a53c2dc1-5fa4-493e-b251-7438ccefcc23\") " pod="openshift-multus/multus-additional-cni-plugins-998qs" Jul 16 21:47:34.005599 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.004180 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3ad1637f-5724-4467-a756-2b059ff5e457-os-release\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:34.005599 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.004176 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6e7f0775-368f-4f7b-be25-3bf85b42f973-env-overrides\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:34.005599 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.004206 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3ad1637f-5724-4467-a756-2b059ff5e457-cni-binary-copy\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:34.005599 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.004256 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3ad1637f-5724-4467-a756-2b059ff5e457-multus-daemon-config\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:34.005599 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.004259 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/58e464a1-1e18-4f00-805d-46c3f88ebd51-etc-modprobe-d\") pod \"tuned-lbh7v\" (UID: \"58e464a1-1e18-4f00-805d-46c3f88ebd51\") " pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" Jul 16 21:47:34.005599 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:34.004281 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:47:34.005599 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.004285 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e7f0775-368f-4f7b-be25-3bf85b42f973-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:34.005599 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.004324 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ad1637f-5724-4467-a756-2b059ff5e457-system-cni-dir\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:34.005599 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.004334 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a53c2dc1-5fa4-493e-b251-7438ccefcc23-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-998qs\" (UID: \"a53c2dc1-5fa4-493e-b251-7438ccefcc23\") " pod="openshift-multus/multus-additional-cni-plugins-998qs" Jul 16 21:47:34.005599 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.004361 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e7f0775-368f-4f7b-be25-3bf85b42f973-run-openvswitch\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:34.005599 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:34.004389 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c33b5633-01c6-4f48-8cb7-6969a24fe86b-metrics-certs podName:c33b5633-01c6-4f48-8cb7-6969a24fe86b nodeName:}" failed. No retries permitted until 2026-07-16 21:47:34.504353064 +0000 UTC m=+3.126796000 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c33b5633-01c6-4f48-8cb7-6969a24fe86b-metrics-certs") pod "network-metrics-daemon-lcl48" (UID: "c33b5633-01c6-4f48-8cb7-6969a24fe86b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:47:34.005599 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.004412 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3ad1637f-5724-4467-a756-2b059ff5e457-host-run-netns\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:34.005599 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.004454 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ad1637f-5724-4467-a756-2b059ff5e457-etc-kubernetes\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:34.005599 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.004518 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e9b9235a-c4fc-4aac-93b4-2964356667f1-kubelet-config\") pod \"global-pull-secret-syncer-jwlrc\" (UID: \"e9b9235a-c4fc-4aac-93b4-2964356667f1\") " pod="kube-system/global-pull-secret-syncer-jwlrc" Jul 16 21:47:34.005599 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.004568 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3ad1637f-5724-4467-a756-2b059ff5e457-host-run-netns\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:34.005599 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.004607 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ad1637f-5724-4467-a756-2b059ff5e457-etc-kubernetes\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:34.006225 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.004639 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a53c2dc1-5fa4-493e-b251-7438ccefcc23-os-release\") pod \"multus-additional-cni-plugins-998qs\" (UID: \"a53c2dc1-5fa4-493e-b251-7438ccefcc23\") " pod="openshift-multus/multus-additional-cni-plugins-998qs" Jul 16 21:47:34.006225 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.004635 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58e464a1-1e18-4f00-805d-46c3f88ebd51-host\") pod \"tuned-lbh7v\" (UID: \"58e464a1-1e18-4f00-805d-46c3f88ebd51\") " pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" Jul 16 21:47:34.006225 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.004673 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/58e464a1-1e18-4f00-805d-46c3f88ebd51-etc-sysconfig\") pod \"tuned-lbh7v\" (UID: \"58e464a1-1e18-4f00-805d-46c3f88ebd51\") " pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" Jul 16 21:47:34.006225 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.004701 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/43f80a88-5e25-4f60-980a-8bb22936ba6c-agent-certs\") pod \"konnectivity-agent-lfwng\" (UID: \"43f80a88-5e25-4f60-980a-8bb22936ba6c\") " pod="kube-system/konnectivity-agent-lfwng" Jul 16 21:47:34.006225 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.004730 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4hrg\" (UniqueName: \"kubernetes.io/projected/f79c076a-55a4-487c-a763-9a9923ba4129-kube-api-access-x4hrg\") pod \"network-check-target-zvq8m\" (UID: \"f79c076a-55a4-487c-a763-9a9923ba4129\") " pod="openshift-network-diagnostics/network-check-target-zvq8m" Jul 16 21:47:34.006225 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.004737 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3ad1637f-5724-4467-a756-2b059ff5e457-os-release\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:34.006225 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.004760 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a53c2dc1-5fa4-493e-b251-7438ccefcc23-tuning-conf-dir\") pod \"multus-additional-cni-plugins-998qs\" (UID: \"a53c2dc1-5fa4-493e-b251-7438ccefcc23\") " pod="openshift-multus/multus-additional-cni-plugins-998qs" Jul 16 21:47:34.006225 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.004786 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58e464a1-1e18-4f00-805d-46c3f88ebd51-sys\") pod \"tuned-lbh7v\" (UID: \"58e464a1-1e18-4f00-805d-46c3f88ebd51\") " pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" Jul 16 21:47:34.006225 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.004817 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a53c2dc1-5fa4-493e-b251-7438ccefcc23-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-998qs\" (UID: \"a53c2dc1-5fa4-493e-b251-7438ccefcc23\") " pod="openshift-multus/multus-additional-cni-plugins-998qs" Jul 16 21:47:34.006225 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.004845 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e7f0775-368f-4f7b-be25-3bf85b42f973-etc-openvswitch\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:34.006225 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.004873 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/58e464a1-1e18-4f00-805d-46c3f88ebd51-etc-sysconfig\") pod \"tuned-lbh7v\" (UID: \"58e464a1-1e18-4f00-805d-46c3f88ebd51\") " pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" Jul 16 21:47:34.006225 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.004882 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6e7f0775-368f-4f7b-be25-3bf85b42f973-host-cni-bin\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:34.006225 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.004917 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6e7f0775-368f-4f7b-be25-3bf85b42f973-host-cni-bin\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:34.006225 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.004943 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58e464a1-1e18-4f00-805d-46c3f88ebd51-sys\") pod \"tuned-lbh7v\" (UID: \"58e464a1-1e18-4f00-805d-46c3f88ebd51\") " pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" Jul 16 21:47:34.006225 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.004997 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a53c2dc1-5fa4-493e-b251-7438ccefcc23-os-release\") pod \"multus-additional-cni-plugins-998qs\" (UID: \"a53c2dc1-5fa4-493e-b251-7438ccefcc23\") " pod="openshift-multus/multus-additional-cni-plugins-998qs" Jul 16 21:47:34.006225 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.005178 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a53c2dc1-5fa4-493e-b251-7438ccefcc23-tuning-conf-dir\") pod \"multus-additional-cni-plugins-998qs\" (UID: \"a53c2dc1-5fa4-493e-b251-7438ccefcc23\") " pod="openshift-multus/multus-additional-cni-plugins-998qs" Jul 16 21:47:34.006225 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.005413 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3ad1637f-5724-4467-a756-2b059ff5e457-cni-binary-copy\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:34.007123 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.005438 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6e7f0775-368f-4f7b-be25-3bf85b42f973-ovnkube-config\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:34.007123 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.005500 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3ad1637f-5724-4467-a756-2b059ff5e457-hostroot\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:34.007123 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.005534 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/58e464a1-1e18-4f00-805d-46c3f88ebd51-etc-sysctl-conf\") pod \"tuned-lbh7v\" (UID: \"58e464a1-1e18-4f00-805d-46c3f88ebd51\") " pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" Jul 16 21:47:34.007123 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.005561 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58e464a1-1e18-4f00-805d-46c3f88ebd51-var-lib-kubelet\") pod \"tuned-lbh7v\" (UID: \"58e464a1-1e18-4f00-805d-46c3f88ebd51\") " pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" Jul 16 21:47:34.007123 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.005592 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/58e464a1-1e18-4f00-805d-46c3f88ebd51-etc-tuned\") pod \"tuned-lbh7v\" (UID: \"58e464a1-1e18-4f00-805d-46c3f88ebd51\") " pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" Jul 16 21:47:34.007123 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.005621 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3ad1637f-5724-4467-a756-2b059ff5e457-host-run-k8s-cni-cncf-io\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:34.007123 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.005653 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e7f0775-368f-4f7b-be25-3bf85b42f973-etc-openvswitch\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:34.007123 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.005726 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58e464a1-1e18-4f00-805d-46c3f88ebd51-etc-kubernetes\") pod \"tuned-lbh7v\" (UID: \"58e464a1-1e18-4f00-805d-46c3f88ebd51\") " pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" Jul 16 21:47:34.007123 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.005760 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e9b9235a-c4fc-4aac-93b4-2964356667f1-dbus\") pod \"global-pull-secret-syncer-jwlrc\" (UID: \"e9b9235a-c4fc-4aac-93b4-2964356667f1\") " pod="kube-system/global-pull-secret-syncer-jwlrc" Jul 16 21:47:34.007123 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.005769 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e7f0775-368f-4f7b-be25-3bf85b42f973-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:34.007123 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.005790 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6e7f0775-368f-4f7b-be25-3bf85b42f973-ovnkube-script-lib\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:34.007123 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.005825 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3ad1637f-5724-4467-a756-2b059ff5e457-host-run-multus-certs\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:34.007123 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.005875 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a53c2dc1-5fa4-493e-b251-7438ccefcc23-system-cni-dir\") pod \"multus-additional-cni-plugins-998qs\" (UID: \"a53c2dc1-5fa4-493e-b251-7438ccefcc23\") " pod="openshift-multus/multus-additional-cni-plugins-998qs" Jul 16 21:47:34.007123 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.005914 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3ad1637f-5724-4467-a756-2b059ff5e457-multus-daemon-config\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:34.007123 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.005916 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6e7f0775-368f-4f7b-be25-3bf85b42f973-node-log\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:34.007123 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.005940 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3ad1637f-5724-4467-a756-2b059ff5e457-host-run-k8s-cni-cncf-io\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:34.007123 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.005964 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6e7f0775-368f-4f7b-be25-3bf85b42f973-run-ovn\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:34.007123 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.005959 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6e7f0775-368f-4f7b-be25-3bf85b42f973-log-socket\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:34.008090 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.006014 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3ad1637f-5724-4467-a756-2b059ff5e457-host-var-lib-cni-bin\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:34.008090 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:34.006047 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 21:47:34.008090 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.006069 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0b72c656-880f-40d7-936f-4f46dc9d9128-kubelet-dir\") pod \"aws-ebs-csi-driver-node-djdlj\" (UID: \"0b72c656-880f-40d7-936f-4f46dc9d9128\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djdlj" Jul 16 21:47:34.008090 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:34.006083 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9b9235a-c4fc-4aac-93b4-2964356667f1-original-pull-secret podName:e9b9235a-c4fc-4aac-93b4-2964356667f1 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:34.50607047 +0000 UTC m=+3.128513403 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9b9235a-c4fc-4aac-93b4-2964356667f1-original-pull-secret") pod "global-pull-secret-syncer-jwlrc" (UID: "e9b9235a-c4fc-4aac-93b4-2964356667f1") : object "kube-system"/"original-pull-secret" not registered Jul 16 21:47:34.008090 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.006116 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/58e464a1-1e18-4f00-805d-46c3f88ebd51-etc-systemd\") pod \"tuned-lbh7v\" (UID: \"58e464a1-1e18-4f00-805d-46c3f88ebd51\") " pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" Jul 16 21:47:34.008090 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.006118 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58e464a1-1e18-4f00-805d-46c3f88ebd51-var-lib-kubelet\") pod \"tuned-lbh7v\" (UID: \"58e464a1-1e18-4f00-805d-46c3f88ebd51\") " pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" Jul 16 21:47:34.008090 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.006165 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ad1637f-5724-4467-a756-2b059ff5e457-host-var-lib-kubelet\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:34.008090 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.006219 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3ad1637f-5724-4467-a756-2b059ff5e457-hostroot\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:34.008090 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.006268 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3ad1637f-5724-4467-a756-2b059ff5e457-multus-socket-dir-parent\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:34.008090 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.006329 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/58e464a1-1e18-4f00-805d-46c3f88ebd51-etc-sysctl-conf\") pod \"tuned-lbh7v\" (UID: \"58e464a1-1e18-4f00-805d-46c3f88ebd51\") " pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" Jul 16 21:47:34.008090 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.006336 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e9b9235a-c4fc-4aac-93b4-2964356667f1-dbus\") pod \"global-pull-secret-syncer-jwlrc\" (UID: \"e9b9235a-c4fc-4aac-93b4-2964356667f1\") " pod="kube-system/global-pull-secret-syncer-jwlrc" Jul 16 21:47:34.008090 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.006364 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6e7f0775-368f-4f7b-be25-3bf85b42f973-systemd-units\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:34.008090 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.006450 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58e464a1-1e18-4f00-805d-46c3f88ebd51-etc-kubernetes\") pod \"tuned-lbh7v\" (UID: \"58e464a1-1e18-4f00-805d-46c3f88ebd51\") " pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" Jul 16 21:47:34.008090 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.006516 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ad1637f-5724-4467-a756-2b059ff5e457-cnibin\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:34.008090 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.006527 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6e7f0775-368f-4f7b-be25-3bf85b42f973-ovnkube-script-lib\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:34.008090 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.007870 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/43f80a88-5e25-4f60-980a-8bb22936ba6c-agent-certs\") pod \"konnectivity-agent-lfwng\" (UID: \"43f80a88-5e25-4f60-980a-8bb22936ba6c\") " pod="kube-system/konnectivity-agent-lfwng" Jul 16 21:47:34.009445 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.009357 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58e464a1-1e18-4f00-805d-46c3f88ebd51-tmp\") pod \"tuned-lbh7v\" (UID: \"58e464a1-1e18-4f00-805d-46c3f88ebd51\") " pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" Jul 16 21:47:34.009968 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.009674 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6e7f0775-368f-4f7b-be25-3bf85b42f973-ovn-node-metrics-cert\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:34.010953 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.010893 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/58e464a1-1e18-4f00-805d-46c3f88ebd51-etc-tuned\") pod \"tuned-lbh7v\" (UID: \"58e464a1-1e18-4f00-805d-46c3f88ebd51\") " pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" Jul 16 21:47:34.015752 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:34.015731 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 21:47:34.015844 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:34.015760 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 21:47:34.015844 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:34.015774 2585 projected.go:196] Error preparing data for projected volume kube-api-access-x4hrg for pod openshift-network-diagnostics/network-check-target-zvq8m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:47:34.015969 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:34.015871 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f79c076a-55a4-487c-a763-9a9923ba4129-kube-api-access-x4hrg podName:f79c076a-55a4-487c-a763-9a9923ba4129 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:34.515853173 +0000 UTC m=+3.138296107 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-x4hrg" (UniqueName: "kubernetes.io/projected/f79c076a-55a4-487c-a763-9a9923ba4129-kube-api-access-x4hrg") pod "network-check-target-zvq8m" (UID: "f79c076a-55a4-487c-a763-9a9923ba4129") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:47:34.016963 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.016583 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5hs4\" (UniqueName: \"kubernetes.io/projected/6e7f0775-368f-4f7b-be25-3bf85b42f973-kube-api-access-j5hs4\") pod \"ovnkube-node-lmrcx\" (UID: \"6e7f0775-368f-4f7b-be25-3bf85b42f973\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:34.017443 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.017318 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww8b5\" (UniqueName: \"kubernetes.io/projected/3ad1637f-5724-4467-a756-2b059ff5e457-kube-api-access-ww8b5\") pod \"multus-ntnh7\" (UID: \"3ad1637f-5724-4467-a756-2b059ff5e457\") " pod="openshift-multus/multus-ntnh7" Jul 16 21:47:34.019581 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.019214 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbr9m\" (UniqueName: \"kubernetes.io/projected/a53c2dc1-5fa4-493e-b251-7438ccefcc23-kube-api-access-qbr9m\") pod \"multus-additional-cni-plugins-998qs\" (UID: \"a53c2dc1-5fa4-493e-b251-7438ccefcc23\") " pod="openshift-multus/multus-additional-cni-plugins-998qs" Jul 16 21:47:34.019581 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.019549 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnfhp\" (UniqueName: \"kubernetes.io/projected/0b72c656-880f-40d7-936f-4f46dc9d9128-kube-api-access-wnfhp\") pod \"aws-ebs-csi-driver-node-djdlj\" (UID: \"0b72c656-880f-40d7-936f-4f46dc9d9128\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djdlj" Jul 16 21:47:34.019875 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.019854 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf8tf\" (UniqueName: \"kubernetes.io/projected/c33b5633-01c6-4f48-8cb7-6969a24fe86b-kube-api-access-vf8tf\") pod \"network-metrics-daemon-lcl48\" (UID: \"c33b5633-01c6-4f48-8cb7-6969a24fe86b\") " pod="openshift-multus/network-metrics-daemon-lcl48" Jul 16 21:47:34.020462 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.020441 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tbd8\" (UniqueName: \"kubernetes.io/projected/58e464a1-1e18-4f00-805d-46c3f88ebd51-kube-api-access-8tbd8\") pod \"tuned-lbh7v\" (UID: \"58e464a1-1e18-4f00-805d-46c3f88ebd51\") " pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" Jul 16 21:47:34.044724 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.044612 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r4g7k" Jul 16 21:47:34.056697 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.056671 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7tltv" Jul 16 21:47:34.059620 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:34.059593 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a9ccc80_b805_462b_87b9_22cfdc7e073c.slice/crio-2ab2fdcf5a3c64e046000743514481d3f95faa7dbc097e95fa8bc15c7b6ce950 WatchSource:0}: Error finding container 2ab2fdcf5a3c64e046000743514481d3f95faa7dbc097e95fa8bc15c7b6ce950: Status 404 returned error can't find the container with id 2ab2fdcf5a3c64e046000743514481d3f95faa7dbc097e95fa8bc15c7b6ce950 Jul 16 21:47:34.066936 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.066709 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-65cgd" Jul 16 21:47:34.067600 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:34.067558 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dabb80a_8529_4289_82e4_e44ca61494c8.slice/crio-c6a77c66eb9e793b1d831a6d09b2d033cba65d3e4277472ea959ba641c3ae93c WatchSource:0}: Error finding container c6a77c66eb9e793b1d831a6d09b2d033cba65d3e4277472ea959ba641c3ae93c: Status 404 returned error can't find the container with id c6a77c66eb9e793b1d831a6d09b2d033cba65d3e4277472ea959ba641c3ae93c Jul 16 21:47:34.074412 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.074378 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lfwng" Jul 16 21:47:34.077244 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:34.077205 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf64092be_22df_4e1c_ab7e_4fcc7afb7ec7.slice/crio-47da790718b61e4fd4df1bd05e2e4e9d0cd8dc6f06bda456e864f14bc8de75c4 WatchSource:0}: Error finding container 47da790718b61e4fd4df1bd05e2e4e9d0cd8dc6f06bda456e864f14bc8de75c4: Status 404 returned error can't find the container with id 47da790718b61e4fd4df1bd05e2e4e9d0cd8dc6f06bda456e864f14bc8de75c4 Jul 16 21:47:34.084540 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.084518 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ntnh7" Jul 16 21:47:34.086863 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:34.086791 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43f80a88_5e25_4f60_980a_8bb22936ba6c.slice/crio-9cc4fe41a0c1ad8d211be474f3c69d185833e000685f6eed88bf023a0cee55b7 WatchSource:0}: Error finding container 9cc4fe41a0c1ad8d211be474f3c69d185833e000685f6eed88bf023a0cee55b7: Status 404 returned error can't find the container with id 9cc4fe41a0c1ad8d211be474f3c69d185833e000685f6eed88bf023a0cee55b7 Jul 16 21:47:34.091803 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.091639 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:47:34.096909 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:34.096884 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ad1637f_5724_4467_a756_2b059ff5e457.slice/crio-c1c28edfbcdc49e2324c150aabe6a36203917e7b5bd243dff91962cc01266fd3 WatchSource:0}: Error finding container c1c28edfbcdc49e2324c150aabe6a36203917e7b5bd243dff91962cc01266fd3: Status 404 returned error can't find the container with id c1c28edfbcdc49e2324c150aabe6a36203917e7b5bd243dff91962cc01266fd3 Jul 16 21:47:34.098849 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.098831 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-998qs" Jul 16 21:47:34.102563 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:34.102542 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e7f0775_368f_4f7b_be25_3bf85b42f973.slice/crio-46a318f66e1c5d482ee04a550478248d0d729f7d4c6cf6321767cc5573264b62 WatchSource:0}: Error finding container 46a318f66e1c5d482ee04a550478248d0d729f7d4c6cf6321767cc5573264b62: Status 404 returned error can't find the container with id 46a318f66e1c5d482ee04a550478248d0d729f7d4c6cf6321767cc5573264b62 Jul 16 21:47:34.108429 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.108409 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djdlj" Jul 16 21:47:34.114453 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.114426 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" Jul 16 21:47:34.509244 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.509204 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c33b5633-01c6-4f48-8cb7-6969a24fe86b-metrics-certs\") pod \"network-metrics-daemon-lcl48\" (UID: \"c33b5633-01c6-4f48-8cb7-6969a24fe86b\") " pod="openshift-multus/network-metrics-daemon-lcl48" Jul 16 21:47:34.509419 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.509251 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9b9235a-c4fc-4aac-93b4-2964356667f1-original-pull-secret\") pod \"global-pull-secret-syncer-jwlrc\" (UID: \"e9b9235a-c4fc-4aac-93b4-2964356667f1\") " pod="kube-system/global-pull-secret-syncer-jwlrc" Jul 16 21:47:34.509419 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:34.509368 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 21:47:34.509537 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:34.509427 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9b9235a-c4fc-4aac-93b4-2964356667f1-original-pull-secret podName:e9b9235a-c4fc-4aac-93b4-2964356667f1 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:35.509410961 +0000 UTC m=+4.131853922 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9b9235a-c4fc-4aac-93b4-2964356667f1-original-pull-secret") pod "global-pull-secret-syncer-jwlrc" (UID: "e9b9235a-c4fc-4aac-93b4-2964356667f1") : object "kube-system"/"original-pull-secret" not registered Jul 16 21:47:34.509537 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:34.509423 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:47:34.509537 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:34.509457 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c33b5633-01c6-4f48-8cb7-6969a24fe86b-metrics-certs podName:c33b5633-01c6-4f48-8cb7-6969a24fe86b nodeName:}" failed. No retries permitted until 2026-07-16 21:47:35.50944851 +0000 UTC m=+4.131891442 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c33b5633-01c6-4f48-8cb7-6969a24fe86b-metrics-certs") pod "network-metrics-daemon-lcl48" (UID: "c33b5633-01c6-4f48-8cb7-6969a24fe86b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:47:34.610509 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.610461 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4hrg\" (UniqueName: \"kubernetes.io/projected/f79c076a-55a4-487c-a763-9a9923ba4129-kube-api-access-x4hrg\") pod \"network-check-target-zvq8m\" (UID: \"f79c076a-55a4-487c-a763-9a9923ba4129\") " pod="openshift-network-diagnostics/network-check-target-zvq8m" Jul 16 21:47:34.610768 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:34.610745 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 21:47:34.610880 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:34.610777 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 21:47:34.610880 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:34.610808 2585 projected.go:196] Error preparing data for projected volume kube-api-access-x4hrg for pod openshift-network-diagnostics/network-check-target-zvq8m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:47:34.610977 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:34.610894 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f79c076a-55a4-487c-a763-9a9923ba4129-kube-api-access-x4hrg podName:f79c076a-55a4-487c-a763-9a9923ba4129 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:35.61086055 +0000 UTC m=+4.233303502 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-x4hrg" (UniqueName: "kubernetes.io/projected/f79c076a-55a4-487c-a763-9a9923ba4129-kube-api-access-x4hrg") pod "network-check-target-zvq8m" (UID: "f79c076a-55a4-487c-a763-9a9923ba4129") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:47:34.807837 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.807745 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 21:42:33 +0000 UTC" deadline="2028-03-10 08:39:00.384573133 +0000 UTC" Jul 16 21:47:34.807837 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.807831 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14458h51m25.576746244s" Jul 16 21:47:34.928575 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.928540 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvq8m" Jul 16 21:47:34.929251 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:34.928695 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvq8m" podUID="f79c076a-55a4-487c-a763-9a9923ba4129" Jul 16 21:47:34.941654 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.941624 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r4g7k" event={"ID":"9a9ccc80-b805-462b-87b9-22cfdc7e073c","Type":"ContainerStarted","Data":"2ab2fdcf5a3c64e046000743514481d3f95faa7dbc097e95fa8bc15c7b6ce950"} Jul 16 21:47:34.942911 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.942881 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" event={"ID":"6e7f0775-368f-4f7b-be25-3bf85b42f973","Type":"ContainerStarted","Data":"46a318f66e1c5d482ee04a550478248d0d729f7d4c6cf6321767cc5573264b62"} Jul 16 21:47:34.944387 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.944353 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ntnh7" event={"ID":"3ad1637f-5724-4467-a756-2b059ff5e457","Type":"ContainerStarted","Data":"c1c28edfbcdc49e2324c150aabe6a36203917e7b5bd243dff91962cc01266fd3"} Jul 16 21:47:34.945729 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.945681 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lfwng" event={"ID":"43f80a88-5e25-4f60-980a-8bb22936ba6c","Type":"ContainerStarted","Data":"9cc4fe41a0c1ad8d211be474f3c69d185833e000685f6eed88bf023a0cee55b7"} Jul 16 21:47:34.947804 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.947779 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-65cgd" event={"ID":"f64092be-22df-4e1c-ab7e-4fcc7afb7ec7","Type":"ContainerStarted","Data":"47da790718b61e4fd4df1bd05e2e4e9d0cd8dc6f06bda456e864f14bc8de75c4"} Jul 16 21:47:34.949090 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:34.949060 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7tltv" event={"ID":"1dabb80a-8529-4289-82e4-e44ca61494c8","Type":"ContainerStarted","Data":"c6a77c66eb9e793b1d831a6d09b2d033cba65d3e4277472ea959ba641c3ae93c"} Jul 16 21:47:35.430464 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:35.430431 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda53c2dc1_5fa4_493e_b251_7438ccefcc23.slice/crio-b2a4fe2169804a90783ac9145eaed1c31d70838523cd68ec12e5ba6ff97cc632 WatchSource:0}: Error finding container b2a4fe2169804a90783ac9145eaed1c31d70838523cd68ec12e5ba6ff97cc632: Status 404 returned error can't find the container with id b2a4fe2169804a90783ac9145eaed1c31d70838523cd68ec12e5ba6ff97cc632 Jul 16 21:47:35.519320 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:35.519271 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c33b5633-01c6-4f48-8cb7-6969a24fe86b-metrics-certs\") pod \"network-metrics-daemon-lcl48\" (UID: \"c33b5633-01c6-4f48-8cb7-6969a24fe86b\") " pod="openshift-multus/network-metrics-daemon-lcl48" Jul 16 21:47:35.519530 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:35.519335 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9b9235a-c4fc-4aac-93b4-2964356667f1-original-pull-secret\") pod \"global-pull-secret-syncer-jwlrc\" (UID: \"e9b9235a-c4fc-4aac-93b4-2964356667f1\") " pod="kube-system/global-pull-secret-syncer-jwlrc" Jul 16 21:47:35.519530 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:35.519490 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 21:47:35.519644 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:35.519541 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9b9235a-c4fc-4aac-93b4-2964356667f1-original-pull-secret podName:e9b9235a-c4fc-4aac-93b4-2964356667f1 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:37.519526706 +0000 UTC m=+6.141969653 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9b9235a-c4fc-4aac-93b4-2964356667f1-original-pull-secret") pod "global-pull-secret-syncer-jwlrc" (UID: "e9b9235a-c4fc-4aac-93b4-2964356667f1") : object "kube-system"/"original-pull-secret" not registered Jul 16 21:47:35.519853 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:35.519835 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:47:35.519901 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:35.519879 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c33b5633-01c6-4f48-8cb7-6969a24fe86b-metrics-certs podName:c33b5633-01c6-4f48-8cb7-6969a24fe86b nodeName:}" failed. No retries permitted until 2026-07-16 21:47:37.51986826 +0000 UTC m=+6.142311192 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c33b5633-01c6-4f48-8cb7-6969a24fe86b-metrics-certs") pod "network-metrics-daemon-lcl48" (UID: "c33b5633-01c6-4f48-8cb7-6969a24fe86b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:47:35.620387 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:35.620352 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4hrg\" (UniqueName: \"kubernetes.io/projected/f79c076a-55a4-487c-a763-9a9923ba4129-kube-api-access-x4hrg\") pod \"network-check-target-zvq8m\" (UID: \"f79c076a-55a4-487c-a763-9a9923ba4129\") " pod="openshift-network-diagnostics/network-check-target-zvq8m" Jul 16 21:47:35.620574 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:35.620553 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 21:47:35.620644 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:35.620574 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 21:47:35.620644 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:35.620588 2585 projected.go:196] Error preparing data for projected volume kube-api-access-x4hrg for pod openshift-network-diagnostics/network-check-target-zvq8m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:47:35.620753 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:35.620644 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f79c076a-55a4-487c-a763-9a9923ba4129-kube-api-access-x4hrg podName:f79c076a-55a4-487c-a763-9a9923ba4129 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:37.620625841 +0000 UTC m=+6.243068776 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-x4hrg" (UniqueName: "kubernetes.io/projected/f79c076a-55a4-487c-a763-9a9923ba4129-kube-api-access-x4hrg") pod "network-check-target-zvq8m" (UID: "f79c076a-55a4-487c-a763-9a9923ba4129") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:47:35.808865 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:35.808825 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 21:42:33 +0000 UTC" deadline="2028-03-28 12:53:24.976947202 +0000 UTC" Jul 16 21:47:35.808865 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:35.808865 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14895h5m49.168087149s" Jul 16 21:47:35.931651 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:35.931621 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcl48" Jul 16 21:47:35.932081 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:35.931767 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcl48" podUID="c33b5633-01c6-4f48-8cb7-6969a24fe86b" Jul 16 21:47:35.932081 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:35.931834 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwlrc" Jul 16 21:47:35.932081 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:35.931907 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwlrc" podUID="e9b9235a-c4fc-4aac-93b4-2964356667f1" Jul 16 21:47:35.952049 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:35.952014 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-998qs" event={"ID":"a53c2dc1-5fa4-493e-b251-7438ccefcc23","Type":"ContainerStarted","Data":"b2a4fe2169804a90783ac9145eaed1c31d70838523cd68ec12e5ba6ff97cc632"} Jul 16 21:47:36.244266 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:36.244235 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58e464a1_1e18_4f00_805d_46c3f88ebd51.slice/crio-341f7ddc7629fa6041198bbc8ef37f115442eefda0367b1a1e616ec1ce55073f WatchSource:0}: Error finding container 341f7ddc7629fa6041198bbc8ef37f115442eefda0367b1a1e616ec1ce55073f: Status 404 returned error can't find the container with id 341f7ddc7629fa6041198bbc8ef37f115442eefda0367b1a1e616ec1ce55073f Jul 16 21:47:36.929300 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:36.929259 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvq8m" Jul 16 21:47:36.929479 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:36.929387 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvq8m" podUID="f79c076a-55a4-487c-a763-9a9923ba4129" Jul 16 21:47:36.955616 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:36.955581 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" event={"ID":"58e464a1-1e18-4f00-805d-46c3f88ebd51","Type":"ContainerStarted","Data":"341f7ddc7629fa6041198bbc8ef37f115442eefda0367b1a1e616ec1ce55073f"} Jul 16 21:47:37.273087 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:47:37.273048 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b72c656_880f_40d7_936f_4f46dc9d9128.slice/crio-a4d7d653c8de36dc1f81d50f02a1a9ff320ffdc6fd4e90bc2f3a880f8448de20 WatchSource:0}: Error finding container a4d7d653c8de36dc1f81d50f02a1a9ff320ffdc6fd4e90bc2f3a880f8448de20: Status 404 returned error can't find the container with id a4d7d653c8de36dc1f81d50f02a1a9ff320ffdc6fd4e90bc2f3a880f8448de20 Jul 16 21:47:37.535899 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:37.535784 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c33b5633-01c6-4f48-8cb7-6969a24fe86b-metrics-certs\") pod \"network-metrics-daemon-lcl48\" (UID: \"c33b5633-01c6-4f48-8cb7-6969a24fe86b\") " pod="openshift-multus/network-metrics-daemon-lcl48" Jul 16 21:47:37.535899 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:37.535842 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9b9235a-c4fc-4aac-93b4-2964356667f1-original-pull-secret\") pod \"global-pull-secret-syncer-jwlrc\" (UID: \"e9b9235a-c4fc-4aac-93b4-2964356667f1\") " pod="kube-system/global-pull-secret-syncer-jwlrc" Jul 16 21:47:37.536104 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:37.536017 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 21:47:37.536104 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:37.536081 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9b9235a-c4fc-4aac-93b4-2964356667f1-original-pull-secret podName:e9b9235a-c4fc-4aac-93b4-2964356667f1 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:41.536061681 +0000 UTC m=+10.158504617 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9b9235a-c4fc-4aac-93b4-2964356667f1-original-pull-secret") pod "global-pull-secret-syncer-jwlrc" (UID: "e9b9235a-c4fc-4aac-93b4-2964356667f1") : object "kube-system"/"original-pull-secret" not registered Jul 16 21:47:37.536506 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:37.536485 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:47:37.536632 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:37.536543 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c33b5633-01c6-4f48-8cb7-6969a24fe86b-metrics-certs podName:c33b5633-01c6-4f48-8cb7-6969a24fe86b nodeName:}" failed. No retries permitted until 2026-07-16 21:47:41.536527917 +0000 UTC m=+10.158970863 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c33b5633-01c6-4f48-8cb7-6969a24fe86b-metrics-certs") pod "network-metrics-daemon-lcl48" (UID: "c33b5633-01c6-4f48-8cb7-6969a24fe86b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:47:37.637306 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:37.637267 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4hrg\" (UniqueName: \"kubernetes.io/projected/f79c076a-55a4-487c-a763-9a9923ba4129-kube-api-access-x4hrg\") pod \"network-check-target-zvq8m\" (UID: \"f79c076a-55a4-487c-a763-9a9923ba4129\") " pod="openshift-network-diagnostics/network-check-target-zvq8m" Jul 16 21:47:37.637472 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:37.637451 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 21:47:37.637472 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:37.637473 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 21:47:37.637579 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:37.637486 2585 projected.go:196] Error preparing data for projected volume kube-api-access-x4hrg for pod openshift-network-diagnostics/network-check-target-zvq8m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:47:37.637579 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:37.637547 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f79c076a-55a4-487c-a763-9a9923ba4129-kube-api-access-x4hrg podName:f79c076a-55a4-487c-a763-9a9923ba4129 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:41.637529815 +0000 UTC m=+10.259972749 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-x4hrg" (UniqueName: "kubernetes.io/projected/f79c076a-55a4-487c-a763-9a9923ba4129-kube-api-access-x4hrg") pod "network-check-target-zvq8m" (UID: "f79c076a-55a4-487c-a763-9a9923ba4129") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:47:37.930523 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:37.929953 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcl48" Jul 16 21:47:37.930523 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:37.930126 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcl48" podUID="c33b5633-01c6-4f48-8cb7-6969a24fe86b" Jul 16 21:47:37.930994 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:37.930834 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwlrc" Jul 16 21:47:37.930994 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:37.930953 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwlrc" podUID="e9b9235a-c4fc-4aac-93b4-2964356667f1" Jul 16 21:47:37.960561 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:37.960525 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djdlj" event={"ID":"0b72c656-880f-40d7-936f-4f46dc9d9128","Type":"ContainerStarted","Data":"a4d7d653c8de36dc1f81d50f02a1a9ff320ffdc6fd4e90bc2f3a880f8448de20"} Jul 16 21:47:38.929092 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:38.929058 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvq8m" Jul 16 21:47:38.929308 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:38.929271 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvq8m" podUID="f79c076a-55a4-487c-a763-9a9923ba4129" Jul 16 21:47:39.929475 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:39.929360 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcl48" Jul 16 21:47:39.929475 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:39.929396 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwlrc" Jul 16 21:47:39.929906 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:39.929513 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcl48" podUID="c33b5633-01c6-4f48-8cb7-6969a24fe86b" Jul 16 21:47:39.929940 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:39.929906 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwlrc" podUID="e9b9235a-c4fc-4aac-93b4-2964356667f1" Jul 16 21:47:39.967944 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:39.967912 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lfwng" event={"ID":"43f80a88-5e25-4f60-980a-8bb22936ba6c","Type":"ContainerStarted","Data":"f2c51e3120a9956d7f269d421ebf102327446044e61c906d5e2b9b2dbf5324ba"} Jul 16 21:47:39.973793 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:39.973437 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-65cgd" event={"ID":"f64092be-22df-4e1c-ab7e-4fcc7afb7ec7","Type":"ContainerStarted","Data":"8f282374cd10b1a91a758e4f0444215f0a2eaa24f85c4dd91c24570f678a6c24"} Jul 16 21:47:39.977135 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:39.977112 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7tltv" event={"ID":"1dabb80a-8529-4289-82e4-e44ca61494c8","Type":"ContainerStarted","Data":"cd6000cd45ec6c2f831b34451e00800129f5f4407f38f8d077476814721153d0"} Jul 16 21:47:39.979862 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:39.979698 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-998qs" event={"ID":"a53c2dc1-5fa4-493e-b251-7438ccefcc23","Type":"ContainerStarted","Data":"7cb53df4b59c0bb4c5825abe0835ab1222f3eb4049195873908b808d725306c2"} Jul 16 21:47:39.982798 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:39.982772 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-255.ec2.internal" event={"ID":"9c641fdc20deb5fa481c19356c3eeb80","Type":"ContainerStarted","Data":"15658eb80a79db057398acb7df628e29a587dd1fb901499e931fd64701ccb80d"} Jul 16 21:47:39.984764 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:39.984739 2585 generic.go:356] "Generic (PLEG): container finished" podID="227134a40e81cd87d8542159249ce174" containerID="9b609d9410e9bfdff9c9b7ca2acd080b6281e804a56d88462389e5d24a494e2b" exitCode=0 Jul 16 21:47:39.984862 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:39.984779 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-255.ec2.internal" event={"ID":"227134a40e81cd87d8542159249ce174","Type":"ContainerDied","Data":"9b609d9410e9bfdff9c9b7ca2acd080b6281e804a56d88462389e5d24a494e2b"} Jul 16 21:47:39.989157 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:39.988249 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lfwng" podStartSLOduration=3.093574537 podStartE2EDuration="7.988234913s" totalImagesPullingTime="4.894660376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:47:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:39.988139226 +0000 UTC m=+8.610582179" watchObservedRunningTime="2026-07-16 21:47:39.988234913 +0000 UTC m=+8.610677872" Jul 16 21:47:40.028139 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:40.027868 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-7tltv" podStartSLOduration=3.049997508 podStartE2EDuration="8.027853193s" totalImagesPullingTime="4.977855685s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:47:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:40.006817396 +0000 UTC m=+8.629260350" watchObservedRunningTime="2026-07-16 21:47:40.027853193 +0000 UTC m=+8.650296183" Jul 16 21:47:40.073442 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:40.073385 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-65cgd" podStartSLOduration=3.1724805050000002 podStartE2EDuration="8.073363487s" totalImagesPullingTime="4.900882982s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:47:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:40.049096458 +0000 UTC m=+8.671539413" watchObservedRunningTime="2026-07-16 21:47:40.073363487 +0000 UTC m=+8.695806441" Jul 16 21:47:40.106882 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:40.106688 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-255.ec2.internal" podStartSLOduration=7.1066662 podStartE2EDuration="7.1066662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:40.103422554 +0000 UTC m=+8.725865506" watchObservedRunningTime="2026-07-16 21:47:40.1066662 +0000 UTC m=+8.729109154" Jul 16 21:47:40.929037 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:40.929003 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvq8m" Jul 16 21:47:40.929193 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:40.929130 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvq8m" podUID="f79c076a-55a4-487c-a763-9a9923ba4129" Jul 16 21:47:40.990958 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:40.990927 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r4g7k" event={"ID":"9a9ccc80-b805-462b-87b9-22cfdc7e073c","Type":"ContainerStarted","Data":"8c39acaaa3367ea1fcdcd192fde7d2b670ec3c71a9740f14ca94d20c83f0c1d2"} Jul 16 21:47:41.575136 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:41.574373 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c33b5633-01c6-4f48-8cb7-6969a24fe86b-metrics-certs\") pod \"network-metrics-daemon-lcl48\" (UID: \"c33b5633-01c6-4f48-8cb7-6969a24fe86b\") " pod="openshift-multus/network-metrics-daemon-lcl48" Jul 16 21:47:41.575136 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:41.574437 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9b9235a-c4fc-4aac-93b4-2964356667f1-original-pull-secret\") pod \"global-pull-secret-syncer-jwlrc\" (UID: \"e9b9235a-c4fc-4aac-93b4-2964356667f1\") " pod="kube-system/global-pull-secret-syncer-jwlrc" Jul 16 21:47:41.575136 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:41.574602 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 21:47:41.575136 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:41.574662 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9b9235a-c4fc-4aac-93b4-2964356667f1-original-pull-secret podName:e9b9235a-c4fc-4aac-93b4-2964356667f1 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:49.574642571 +0000 UTC m=+18.197085505 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9b9235a-c4fc-4aac-93b4-2964356667f1-original-pull-secret") pod "global-pull-secret-syncer-jwlrc" (UID: "e9b9235a-c4fc-4aac-93b4-2964356667f1") : object "kube-system"/"original-pull-secret" not registered Jul 16 21:47:41.575136 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:41.575050 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:47:41.575136 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:41.575101 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c33b5633-01c6-4f48-8cb7-6969a24fe86b-metrics-certs podName:c33b5633-01c6-4f48-8cb7-6969a24fe86b nodeName:}" failed. No retries permitted until 2026-07-16 21:47:49.575085337 +0000 UTC m=+18.197528291 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c33b5633-01c6-4f48-8cb7-6969a24fe86b-metrics-certs") pod "network-metrics-daemon-lcl48" (UID: "c33b5633-01c6-4f48-8cb7-6969a24fe86b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:47:41.675997 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:41.675376 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4hrg\" (UniqueName: \"kubernetes.io/projected/f79c076a-55a4-487c-a763-9a9923ba4129-kube-api-access-x4hrg\") pod \"network-check-target-zvq8m\" (UID: \"f79c076a-55a4-487c-a763-9a9923ba4129\") " pod="openshift-network-diagnostics/network-check-target-zvq8m" Jul 16 21:47:41.675997 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:41.675582 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 21:47:41.675997 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:41.675603 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 21:47:41.675997 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:41.675616 2585 projected.go:196] Error preparing data for projected volume kube-api-access-x4hrg for pod openshift-network-diagnostics/network-check-target-zvq8m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:47:41.675997 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:41.675674 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f79c076a-55a4-487c-a763-9a9923ba4129-kube-api-access-x4hrg podName:f79c076a-55a4-487c-a763-9a9923ba4129 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:49.675654887 +0000 UTC m=+18.298097833 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-x4hrg" (UniqueName: "kubernetes.io/projected/f79c076a-55a4-487c-a763-9a9923ba4129-kube-api-access-x4hrg") pod "network-check-target-zvq8m" (UID: "f79c076a-55a4-487c-a763-9a9923ba4129") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:47:41.914864 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:41.914781 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lfwng" Jul 16 21:47:41.929371 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:41.929345 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcl48" Jul 16 21:47:41.929500 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:41.929436 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcl48" podUID="c33b5633-01c6-4f48-8cb7-6969a24fe86b" Jul 16 21:47:41.929940 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:41.929787 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwlrc" Jul 16 21:47:41.929940 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:41.929899 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwlrc" podUID="e9b9235a-c4fc-4aac-93b4-2964356667f1" Jul 16 21:47:41.995098 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:41.995066 2585 generic.go:356] "Generic (PLEG): container finished" podID="a53c2dc1-5fa4-493e-b251-7438ccefcc23" containerID="7cb53df4b59c0bb4c5825abe0835ab1222f3eb4049195873908b808d725306c2" exitCode=0 Jul 16 21:47:41.995636 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:41.995600 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-998qs" event={"ID":"a53c2dc1-5fa4-493e-b251-7438ccefcc23","Type":"ContainerDied","Data":"7cb53df4b59c0bb4c5825abe0835ab1222f3eb4049195873908b808d725306c2"} Jul 16 21:47:42.025496 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:42.025421 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-r4g7k" podStartSLOduration=5.111596602 podStartE2EDuration="10.025401469s" totalImagesPullingTime="4.913804867s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:47:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:41.019716945 +0000 UTC m=+9.642159899" watchObservedRunningTime="2026-07-16 21:47:42.025401469 +0000 UTC m=+10.647844423" Jul 16 21:47:42.587187 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:42.587144 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lfwng" Jul 16 21:47:42.587713 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:42.587691 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lfwng" Jul 16 21:47:42.929553 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:42.929470 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvq8m" Jul 16 21:47:42.929686 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:42.929606 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvq8m" podUID="f79c076a-55a4-487c-a763-9a9923ba4129" Jul 16 21:47:42.997684 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:42.997653 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lfwng" Jul 16 21:47:43.928869 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:43.928835 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcl48" Jul 16 21:47:43.929049 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:43.928835 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwlrc" Jul 16 21:47:43.929049 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:43.928972 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcl48" podUID="c33b5633-01c6-4f48-8cb7-6969a24fe86b" Jul 16 21:47:43.929170 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:43.929060 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwlrc" podUID="e9b9235a-c4fc-4aac-93b4-2964356667f1" Jul 16 21:47:44.929566 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:44.929312 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvq8m" Jul 16 21:47:44.929989 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:44.929592 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvq8m" podUID="f79c076a-55a4-487c-a763-9a9923ba4129" Jul 16 21:47:45.928725 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:45.928691 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcl48" Jul 16 21:47:45.928891 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:45.928692 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwlrc" Jul 16 21:47:45.928891 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:45.928817 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcl48" podUID="c33b5633-01c6-4f48-8cb7-6969a24fe86b" Jul 16 21:47:45.928996 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:45.928882 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwlrc" podUID="e9b9235a-c4fc-4aac-93b4-2964356667f1" Jul 16 21:47:46.929373 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:46.929344 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvq8m" Jul 16 21:47:46.929756 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:46.929440 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvq8m" podUID="f79c076a-55a4-487c-a763-9a9923ba4129" Jul 16 21:47:47.929280 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:47.929234 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcl48" Jul 16 21:47:47.929280 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:47.929278 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwlrc" Jul 16 21:47:47.929766 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:47.929371 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcl48" podUID="c33b5633-01c6-4f48-8cb7-6969a24fe86b" Jul 16 21:47:47.929766 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:47.929517 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwlrc" podUID="e9b9235a-c4fc-4aac-93b4-2964356667f1" Jul 16 21:47:48.928743 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:48.928707 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvq8m" Jul 16 21:47:48.928952 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:48.928841 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvq8m" podUID="f79c076a-55a4-487c-a763-9a9923ba4129" Jul 16 21:47:49.634274 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:49.634234 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c33b5633-01c6-4f48-8cb7-6969a24fe86b-metrics-certs\") pod \"network-metrics-daemon-lcl48\" (UID: \"c33b5633-01c6-4f48-8cb7-6969a24fe86b\") " pod="openshift-multus/network-metrics-daemon-lcl48" Jul 16 21:47:49.634728 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:49.634290 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9b9235a-c4fc-4aac-93b4-2964356667f1-original-pull-secret\") pod \"global-pull-secret-syncer-jwlrc\" (UID: \"e9b9235a-c4fc-4aac-93b4-2964356667f1\") " pod="kube-system/global-pull-secret-syncer-jwlrc" Jul 16 21:47:49.634728 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:49.634442 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 21:47:49.634728 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:49.634513 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9b9235a-c4fc-4aac-93b4-2964356667f1-original-pull-secret podName:e9b9235a-c4fc-4aac-93b4-2964356667f1 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:05.634493438 +0000 UTC m=+34.256936373 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9b9235a-c4fc-4aac-93b4-2964356667f1-original-pull-secret") pod "global-pull-secret-syncer-jwlrc" (UID: "e9b9235a-c4fc-4aac-93b4-2964356667f1") : object "kube-system"/"original-pull-secret" not registered Jul 16 21:47:49.634728 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:49.634533 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:47:49.634728 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:49.634595 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c33b5633-01c6-4f48-8cb7-6969a24fe86b-metrics-certs podName:c33b5633-01c6-4f48-8cb7-6969a24fe86b nodeName:}" failed. No retries permitted until 2026-07-16 21:48:05.634577003 +0000 UTC m=+34.257019958 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c33b5633-01c6-4f48-8cb7-6969a24fe86b-metrics-certs") pod "network-metrics-daemon-lcl48" (UID: "c33b5633-01c6-4f48-8cb7-6969a24fe86b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:47:49.734667 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:49.734630 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4hrg\" (UniqueName: \"kubernetes.io/projected/f79c076a-55a4-487c-a763-9a9923ba4129-kube-api-access-x4hrg\") pod \"network-check-target-zvq8m\" (UID: \"f79c076a-55a4-487c-a763-9a9923ba4129\") " pod="openshift-network-diagnostics/network-check-target-zvq8m" Jul 16 21:47:49.734866 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:49.734844 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 21:47:49.734866 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:49.734865 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 21:47:49.734969 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:49.734878 2585 projected.go:196] Error preparing data for projected volume kube-api-access-x4hrg for pod openshift-network-diagnostics/network-check-target-zvq8m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:47:49.734969 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:49.734931 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f79c076a-55a4-487c-a763-9a9923ba4129-kube-api-access-x4hrg podName:f79c076a-55a4-487c-a763-9a9923ba4129 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:05.734913369 +0000 UTC m=+34.357356320 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-x4hrg" (UniqueName: "kubernetes.io/projected/f79c076a-55a4-487c-a763-9a9923ba4129-kube-api-access-x4hrg") pod "network-check-target-zvq8m" (UID: "f79c076a-55a4-487c-a763-9a9923ba4129") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:47:49.928759 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:49.928678 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwlrc" Jul 16 21:47:49.928926 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:49.928823 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwlrc" podUID="e9b9235a-c4fc-4aac-93b4-2964356667f1" Jul 16 21:47:49.929169 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:49.928678 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcl48" Jul 16 21:47:49.929169 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:49.929133 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcl48" podUID="c33b5633-01c6-4f48-8cb7-6969a24fe86b" Jul 16 21:47:50.928672 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:50.928632 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvq8m" Jul 16 21:47:50.929115 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:50.928759 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvq8m" podUID="f79c076a-55a4-487c-a763-9a9923ba4129" Jul 16 21:47:51.930231 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:51.930187 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcl48" Jul 16 21:47:51.930741 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:51.930338 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcl48" podUID="c33b5633-01c6-4f48-8cb7-6969a24fe86b" Jul 16 21:47:51.930741 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:51.930375 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwlrc" Jul 16 21:47:51.930741 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:51.930455 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwlrc" podUID="e9b9235a-c4fc-4aac-93b4-2964356667f1" Jul 16 21:47:52.928729 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:52.928699 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvq8m" Jul 16 21:47:52.928923 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:52.928820 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvq8m" podUID="f79c076a-55a4-487c-a763-9a9923ba4129" Jul 16 21:47:53.929423 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:53.929391 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwlrc" Jul 16 21:47:53.930137 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:53.929391 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcl48" Jul 16 21:47:53.930137 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:53.929519 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwlrc" podUID="e9b9235a-c4fc-4aac-93b4-2964356667f1" Jul 16 21:47:53.930137 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:53.929604 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcl48" podUID="c33b5633-01c6-4f48-8cb7-6969a24fe86b" Jul 16 21:47:54.019292 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:54.019258 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-255.ec2.internal" event={"ID":"227134a40e81cd87d8542159249ce174","Type":"ContainerStarted","Data":"b81b1ced40466ecc45223818e0d0eda88073950585ae526120a5929c35910f95"} Jul 16 21:47:54.020875 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:54.020845 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djdlj" event={"ID":"0b72c656-880f-40d7-936f-4f46dc9d9128","Type":"ContainerStarted","Data":"40a42acbf67653263862e6fa3da3af0d06a6ae8919ab1e8861479f771127fa2e"} Jul 16 21:47:54.035334 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:54.035284 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-255.ec2.internal" podStartSLOduration=21.035266386 podStartE2EDuration="21.035266386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:54.034736943 +0000 UTC m=+22.657179900" watchObservedRunningTime="2026-07-16 21:47:54.035266386 +0000 UTC m=+22.657709341" Jul 16 21:47:54.928609 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:54.928580 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvq8m" Jul 16 21:47:54.928840 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:54.928677 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvq8m" podUID="f79c076a-55a4-487c-a763-9a9923ba4129" Jul 16 21:47:55.834578 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:55.834420 2585 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 16 21:47:55.928912 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:55.928760 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcl48" Jul 16 21:47:55.929010 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:55.928811 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwlrc" Jul 16 21:47:55.929072 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:55.929014 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcl48" podUID="c33b5633-01c6-4f48-8cb7-6969a24fe86b" Jul 16 21:47:55.929130 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:55.929104 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwlrc" podUID="e9b9235a-c4fc-4aac-93b4-2964356667f1" Jul 16 21:47:56.027139 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:56.027114 2585 generic.go:356] "Generic (PLEG): container finished" podID="a53c2dc1-5fa4-493e-b251-7438ccefcc23" containerID="b4a8e689100e7723b244f9dde46dd454c6c8f01b7ab4908f592949eceb661e8d" exitCode=0 Jul 16 21:47:56.027282 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:56.027186 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-998qs" event={"ID":"a53c2dc1-5fa4-493e-b251-7438ccefcc23","Type":"ContainerDied","Data":"b4a8e689100e7723b244f9dde46dd454c6c8f01b7ab4908f592949eceb661e8d"} Jul 16 21:47:56.028886 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:56.028838 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" event={"ID":"58e464a1-1e18-4f00-805d-46c3f88ebd51","Type":"ContainerStarted","Data":"5982c67200b9b8a0f5fbd711e81343cf7e0a9bdfd5ddbe4b3a53108fc0a33296"} Jul 16 21:47:56.030720 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:56.030695 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djdlj" event={"ID":"0b72c656-880f-40d7-936f-4f46dc9d9128","Type":"ContainerStarted","Data":"b1d83086e92552bf65140a81a01cdb03e5ec28db1fe98e436474877c83c78b79"} Jul 16 21:47:56.033767 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:56.033745 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" event={"ID":"6e7f0775-368f-4f7b-be25-3bf85b42f973","Type":"ContainerStarted","Data":"17b10f48ebd7e477a867fa30799dcbcddb41483d405626f854e2bc36376eb61b"} Jul 16 21:47:56.033867 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:56.033817 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" event={"ID":"6e7f0775-368f-4f7b-be25-3bf85b42f973","Type":"ContainerStarted","Data":"0b05fdb10e6f7c5a6b89a58e8a442b31f22f19077eb166d9be9094eed91db372"} Jul 16 21:47:56.033965 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:56.033868 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" event={"ID":"6e7f0775-368f-4f7b-be25-3bf85b42f973","Type":"ContainerStarted","Data":"4ec9d752ef1b4939fd1d80c17f1adcb70b893353583cc1103ceece81dfe5f6dc"} Jul 16 21:47:56.033965 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:56.033914 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" event={"ID":"6e7f0775-368f-4f7b-be25-3bf85b42f973","Type":"ContainerStarted","Data":"f7e563acfb91077311df49901c8b7cf2ac77cf22c3243e08a8349b7137f14f6a"} Jul 16 21:47:56.034080 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:56.033963 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" event={"ID":"6e7f0775-368f-4f7b-be25-3bf85b42f973","Type":"ContainerStarted","Data":"1bfa83508213680fc5e3d550037be359a2ffca6c4e2f6537cb1e1ac9e7819426"} Jul 16 21:47:56.034080 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:56.034010 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" event={"ID":"6e7f0775-368f-4f7b-be25-3bf85b42f973","Type":"ContainerStarted","Data":"b61a1598821a3c7b34a0b912828c86a59c6e276ba6f578b01214784281e66295"} Jul 16 21:47:56.035320 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:56.035298 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ntnh7" event={"ID":"3ad1637f-5724-4467-a756-2b059ff5e457","Type":"ContainerStarted","Data":"ce6e0b5c8be34921066257a361d4ddc5d7be60c73e5d386809521b2562901457"} Jul 16 21:47:56.106933 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:56.106885 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ntnh7" podStartSLOduration=5.096786478 podStartE2EDuration="24.106871861s" totalImagesPullingTime="19.010085383s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:47:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:56.07827439 +0000 UTC m=+24.700717345" watchObservedRunningTime="2026-07-16 21:47:56.106871861 +0000 UTC m=+24.729314814" Jul 16 21:47:56.107331 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:56.107308 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lbh7v" podStartSLOduration=7.276471047 podStartE2EDuration="24.107300526s" totalImagesPullingTime="16.830829479s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:47:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:56.106525334 +0000 UTC m=+24.728968288" watchObservedRunningTime="2026-07-16 21:47:56.107300526 +0000 UTC m=+24.729743479" Jul 16 21:47:56.819720 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:56.819639 2585 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T21:47:55.834576495Z","UUID":"f0c810d4-3ca7-4815-822a-4b36e45f61d9","Handler":null,"Name":"","Endpoint":""} Jul 16 21:47:56.821582 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:56.821523 2585 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 16 21:47:56.821582 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:56.821566 2585 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 16 21:47:56.929012 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:56.928991 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvq8m" Jul 16 21:47:56.929346 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:56.929077 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvq8m" podUID="f79c076a-55a4-487c-a763-9a9923ba4129" Jul 16 21:47:57.458305 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:57.458233 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:47:57.684706 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:57.684683 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:47:57.927209 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:57.927182 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:47:57.929274 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:57.929257 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcl48" Jul 16 21:47:57.929657 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:57.929367 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcl48" podUID="c33b5633-01c6-4f48-8cb7-6969a24fe86b" Jul 16 21:47:57.929657 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:57.929395 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwlrc" Jul 16 21:47:57.929657 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:57.929467 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwlrc" podUID="e9b9235a-c4fc-4aac-93b4-2964356667f1" Jul 16 21:47:58.041509 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:58.041485 2585 generic.go:356] "Generic (PLEG): container finished" podID="a53c2dc1-5fa4-493e-b251-7438ccefcc23" containerID="fa63f91b065327e6986bc1f7b626512f1c357cfab18a9bdcb61846fe364ba9d0" exitCode=0 Jul 16 21:47:58.041640 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:58.041556 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-998qs" event={"ID":"a53c2dc1-5fa4-493e-b251-7438ccefcc23","Type":"ContainerDied","Data":"fa63f91b065327e6986bc1f7b626512f1c357cfab18a9bdcb61846fe364ba9d0"} Jul 16 21:47:58.043709 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:58.043688 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djdlj" event={"ID":"0b72c656-880f-40d7-936f-4f46dc9d9128","Type":"ContainerStarted","Data":"2e24e34cd060c297e1ab96442b05ade988350203ecdcc0f8a08572fa638324d9"} Jul 16 21:47:58.105368 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:58.105346 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:47:58.928554 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:58.928535 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvq8m" Jul 16 21:47:58.928700 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:58.928618 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvq8m" podUID="f79c076a-55a4-487c-a763-9a9923ba4129" Jul 16 21:47:59.047720 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:59.047689 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-998qs" event={"ID":"a53c2dc1-5fa4-493e-b251-7438ccefcc23","Type":"ContainerStarted","Data":"8d7817722a2976b7b17706969adb7a120d9a52f650791b2052593e5abd9037a7"} Jul 16 21:47:59.050763 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:59.050741 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" event={"ID":"6e7f0775-368f-4f7b-be25-3bf85b42f973","Type":"ContainerStarted","Data":"dcea611b98a76c9a89e3101d02acdd0413cc9a3cb7a2af354d469ae6f3dce740"} Jul 16 21:47:59.085997 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:59.085957 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-djdlj" podStartSLOduration=7.463238828 podStartE2EDuration="27.085941614s" totalImagesPullingTime="19.622702786s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:47:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:58.124652235 +0000 UTC m=+26.747095182" watchObservedRunningTime="2026-07-16 21:47:59.085941614 +0000 UTC m=+27.708384571" Jul 16 21:47:59.929014 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:59.928979 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcl48" Jul 16 21:47:59.929171 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:47:59.928979 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwlrc" Jul 16 21:47:59.929171 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:59.929078 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcl48" podUID="c33b5633-01c6-4f48-8cb7-6969a24fe86b" Jul 16 21:47:59.929298 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:47:59.929188 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwlrc" podUID="e9b9235a-c4fc-4aac-93b4-2964356667f1" Jul 16 21:48:00.055049 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:00.055018 2585 generic.go:356] "Generic (PLEG): container finished" podID="a53c2dc1-5fa4-493e-b251-7438ccefcc23" containerID="8d7817722a2976b7b17706969adb7a120d9a52f650791b2052593e5abd9037a7" exitCode=0 Jul 16 21:48:00.055602 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:00.055054 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-998qs" event={"ID":"a53c2dc1-5fa4-493e-b251-7438ccefcc23","Type":"ContainerDied","Data":"8d7817722a2976b7b17706969adb7a120d9a52f650791b2052593e5abd9037a7"} Jul 16 21:48:00.928942 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:00.928757 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvq8m" Jul 16 21:48:00.929137 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:00.929024 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvq8m" podUID="f79c076a-55a4-487c-a763-9a9923ba4129" Jul 16 21:48:00.999791 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:00.999756 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:01.061660 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:01.061616 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" event={"ID":"6e7f0775-368f-4f7b-be25-3bf85b42f973","Type":"ContainerStarted","Data":"84436fe83666ccdee7c7c09b53e7ac682f1ebb92012cf74e62c2c3dad63fd583"} Jul 16 21:48:01.062208 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:01.061883 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:48:01.079972 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:01.079951 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:48:01.103974 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:01.103922 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" podStartSLOduration=8.076612176 podStartE2EDuration="29.103903678s" totalImagesPullingTime="21.027291502s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:47:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:01.103283028 +0000 UTC m=+29.725725982" watchObservedRunningTime="2026-07-16 21:48:01.103903678 +0000 UTC m=+29.726346632" Jul 16 21:48:01.111791 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:01.111769 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:01.929414 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:01.929380 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcl48" Jul 16 21:48:01.929599 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:01.929487 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcl48" podUID="c33b5633-01c6-4f48-8cb7-6969a24fe86b" Jul 16 21:48:01.929599 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:01.929547 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwlrc" Jul 16 21:48:01.929710 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:01.929668 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwlrc" podUID="e9b9235a-c4fc-4aac-93b4-2964356667f1" Jul 16 21:48:02.064256 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:02.064204 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:48:02.064696 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:02.064305 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:48:02.080232 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:02.080168 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:48:02.792208 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:02.792064 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lcl48"] Jul 16 21:48:02.792411 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:02.792235 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcl48" Jul 16 21:48:02.792411 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:02.792343 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcl48" podUID="c33b5633-01c6-4f48-8cb7-6969a24fe86b" Jul 16 21:48:02.802055 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:02.802025 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jwlrc"] Jul 16 21:48:02.802196 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:02.802151 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwlrc" Jul 16 21:48:02.802337 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:02.802276 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwlrc" podUID="e9b9235a-c4fc-4aac-93b4-2964356667f1" Jul 16 21:48:02.803263 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:02.803238 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zvq8m"] Jul 16 21:48:02.803366 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:02.803335 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvq8m" Jul 16 21:48:02.803456 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:02.803433 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvq8m" podUID="f79c076a-55a4-487c-a763-9a9923ba4129" Jul 16 21:48:03.069440 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:03.069405 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:03.176120 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:03.176083 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:03.291120 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:03.291073 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:04.928830 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:04.928673 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwlrc" Jul 16 21:48:04.929265 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:04.928707 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvq8m" Jul 16 21:48:04.929265 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:04.928911 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwlrc" podUID="e9b9235a-c4fc-4aac-93b4-2964356667f1" Jul 16 21:48:04.929265 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:04.928711 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcl48" Jul 16 21:48:04.929265 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:04.929004 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvq8m" podUID="f79c076a-55a4-487c-a763-9a9923ba4129" Jul 16 21:48:04.929265 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:04.929097 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcl48" podUID="c33b5633-01c6-4f48-8cb7-6969a24fe86b" Jul 16 21:48:05.072188 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:05.072157 2585 generic.go:356] "Generic (PLEG): container finished" podID="a53c2dc1-5fa4-493e-b251-7438ccefcc23" containerID="bf4458723c6f2675efd524c6517201a43b928edfdddc4ed622902d664b995151" exitCode=0 Jul 16 21:48:05.072337 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:05.072235 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-998qs" event={"ID":"a53c2dc1-5fa4-493e-b251-7438ccefcc23","Type":"ContainerDied","Data":"bf4458723c6f2675efd524c6517201a43b928edfdddc4ed622902d664b995151"} Jul 16 21:48:05.647855 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:05.647822 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c33b5633-01c6-4f48-8cb7-6969a24fe86b-metrics-certs\") pod \"network-metrics-daemon-lcl48\" (UID: \"c33b5633-01c6-4f48-8cb7-6969a24fe86b\") " pod="openshift-multus/network-metrics-daemon-lcl48" Jul 16 21:48:05.647855 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:05.647858 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9b9235a-c4fc-4aac-93b4-2964356667f1-original-pull-secret\") pod \"global-pull-secret-syncer-jwlrc\" (UID: \"e9b9235a-c4fc-4aac-93b4-2964356667f1\") " pod="kube-system/global-pull-secret-syncer-jwlrc" Jul 16 21:48:05.648026 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:05.647956 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 21:48:05.648026 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:05.647957 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:48:05.648026 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:05.648000 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9b9235a-c4fc-4aac-93b4-2964356667f1-original-pull-secret podName:e9b9235a-c4fc-4aac-93b4-2964356667f1 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:37.647988587 +0000 UTC m=+66.270431518 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9b9235a-c4fc-4aac-93b4-2964356667f1-original-pull-secret") pod "global-pull-secret-syncer-jwlrc" (UID: "e9b9235a-c4fc-4aac-93b4-2964356667f1") : object "kube-system"/"original-pull-secret" not registered Jul 16 21:48:05.648026 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:05.648014 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c33b5633-01c6-4f48-8cb7-6969a24fe86b-metrics-certs podName:c33b5633-01c6-4f48-8cb7-6969a24fe86b nodeName:}" failed. No retries permitted until 2026-07-16 21:48:37.648007409 +0000 UTC m=+66.270450341 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c33b5633-01c6-4f48-8cb7-6969a24fe86b-metrics-certs") pod "network-metrics-daemon-lcl48" (UID: "c33b5633-01c6-4f48-8cb7-6969a24fe86b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:48:05.748672 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:05.748646 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4hrg\" (UniqueName: \"kubernetes.io/projected/f79c076a-55a4-487c-a763-9a9923ba4129-kube-api-access-x4hrg\") pod \"network-check-target-zvq8m\" (UID: \"f79c076a-55a4-487c-a763-9a9923ba4129\") " pod="openshift-network-diagnostics/network-check-target-zvq8m" Jul 16 21:48:05.748783 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:05.748772 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 21:48:05.748829 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:05.748788 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 21:48:05.748829 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:05.748797 2585 projected.go:196] Error preparing data for projected volume kube-api-access-x4hrg for pod openshift-network-diagnostics/network-check-target-zvq8m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:48:05.748884 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:05.748838 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f79c076a-55a4-487c-a763-9a9923ba4129-kube-api-access-x4hrg podName:f79c076a-55a4-487c-a763-9a9923ba4129 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:37.748823285 +0000 UTC m=+66.371266218 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-x4hrg" (UniqueName: "kubernetes.io/projected/f79c076a-55a4-487c-a763-9a9923ba4129-kube-api-access-x4hrg") pod "network-check-target-zvq8m" (UID: "f79c076a-55a4-487c-a763-9a9923ba4129") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:48:06.022148 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:06.022075 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:06.077658 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:06.077629 2585 generic.go:356] "Generic (PLEG): container finished" podID="a53c2dc1-5fa4-493e-b251-7438ccefcc23" containerID="664f0d0296ff61503f8a43b29f0b38fe30514c6bdc50c6a0f3aaba0996596847" exitCode=0 Jul 16 21:48:06.077791 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:06.077672 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-998qs" event={"ID":"a53c2dc1-5fa4-493e-b251-7438ccefcc23","Type":"ContainerDied","Data":"664f0d0296ff61503f8a43b29f0b38fe30514c6bdc50c6a0f3aaba0996596847"} Jul 16 21:48:06.134716 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:06.134678 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:06.929110 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:06.929086 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvq8m" Jul 16 21:48:06.929272 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:06.929090 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwlrc" Jul 16 21:48:06.929272 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:06.929177 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvq8m" podUID="f79c076a-55a4-487c-a763-9a9923ba4129" Jul 16 21:48:06.929339 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:06.929274 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwlrc" podUID="e9b9235a-c4fc-4aac-93b4-2964356667f1" Jul 16 21:48:06.929339 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:06.929090 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcl48" Jul 16 21:48:06.929403 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:06.929389 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcl48" podUID="c33b5633-01c6-4f48-8cb7-6969a24fe86b" Jul 16 21:48:07.082720 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:07.082691 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-998qs" event={"ID":"a53c2dc1-5fa4-493e-b251-7438ccefcc23","Type":"ContainerStarted","Data":"a36ec96948cdd493727427fc3354842b5b262d2ac85be79195441c1f64d94144"} Jul 16 21:48:07.115928 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:07.115886 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-998qs" podStartSLOduration=11.524298757 podStartE2EDuration="35.115871212s" totalImagesPullingTime="23.591572455s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:47:32 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:07.115471001 +0000 UTC m=+35.737913953" watchObservedRunningTime="2026-07-16 21:48:07.115871212 +0000 UTC m=+35.738314165" Jul 16 21:48:07.316963 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:07.316935 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:07.414192 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:07.414160 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:07.516840 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:07.516817 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:07.624915 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:07.624862 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:07.737404 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:07.737380 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-255.ec2.internal" event="NodeReady" Jul 16 21:48:07.737524 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:07.737486 2585 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 16 21:48:07.799524 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:07.799503 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5hrrt"] Jul 16 21:48:07.801486 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:07.801471 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5hrrt" Jul 16 21:48:07.804031 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:07.804006 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 16 21:48:07.804124 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:07.804032 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 16 21:48:07.804124 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:07.804099 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9tv9m\"" Jul 16 21:48:07.807352 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:07.807336 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fbvhc"] Jul 16 21:48:07.809016 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:07.809001 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fbvhc" Jul 16 21:48:07.811512 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:07.811495 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 16 21:48:07.811578 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:07.811495 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 16 21:48:07.811731 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:07.811718 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 16 21:48:07.811783 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:07.811763 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fbkjh\"" Jul 16 21:48:07.826506 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:07.826480 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5hrrt"] Jul 16 21:48:07.827004 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:07.826987 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fbvhc"] Jul 16 21:48:07.863773 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:07.863749 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr8ll\" (UniqueName: \"kubernetes.io/projected/7b50691f-e781-4af6-bb30-6600349a399f-kube-api-access-tr8ll\") pod \"dns-default-5hrrt\" (UID: \"7b50691f-e781-4af6-bb30-6600349a399f\") " pod="openshift-dns/dns-default-5hrrt" Jul 16 21:48:07.863872 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:07.863781 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b50691f-e781-4af6-bb30-6600349a399f-tmp-dir\") pod \"dns-default-5hrrt\" (UID: \"7b50691f-e781-4af6-bb30-6600349a399f\") " pod="openshift-dns/dns-default-5hrrt" Jul 16 21:48:07.863872 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:07.863821 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11bd6b74-116b-4233-a3f5-5b12c68be2f9-cert\") pod \"ingress-canary-fbvhc\" (UID: \"11bd6b74-116b-4233-a3f5-5b12c68be2f9\") " pod="openshift-ingress-canary/ingress-canary-fbvhc" Jul 16 21:48:07.863872 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:07.863845 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qccb2\" (UniqueName: \"kubernetes.io/projected/11bd6b74-116b-4233-a3f5-5b12c68be2f9-kube-api-access-qccb2\") pod \"ingress-canary-fbvhc\" (UID: \"11bd6b74-116b-4233-a3f5-5b12c68be2f9\") " pod="openshift-ingress-canary/ingress-canary-fbvhc" Jul 16 21:48:07.863971 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:07.863886 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b50691f-e781-4af6-bb30-6600349a399f-metrics-tls\") pod \"dns-default-5hrrt\" (UID: \"7b50691f-e781-4af6-bb30-6600349a399f\") " pod="openshift-dns/dns-default-5hrrt" Jul 16 21:48:07.863971 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:07.863944 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7b50691f-e781-4af6-bb30-6600349a399f-config-volume\") pod \"dns-default-5hrrt\" (UID: \"7b50691f-e781-4af6-bb30-6600349a399f\") " pod="openshift-dns/dns-default-5hrrt" Jul 16 21:48:07.964393 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:07.964339 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b50691f-e781-4af6-bb30-6600349a399f-tmp-dir\") pod \"dns-default-5hrrt\" (UID: \"7b50691f-e781-4af6-bb30-6600349a399f\") " pod="openshift-dns/dns-default-5hrrt" Jul 16 21:48:07.964393 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:07.964377 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11bd6b74-116b-4233-a3f5-5b12c68be2f9-cert\") pod \"ingress-canary-fbvhc\" (UID: \"11bd6b74-116b-4233-a3f5-5b12c68be2f9\") " pod="openshift-ingress-canary/ingress-canary-fbvhc" Jul 16 21:48:07.964533 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:07.964397 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qccb2\" (UniqueName: \"kubernetes.io/projected/11bd6b74-116b-4233-a3f5-5b12c68be2f9-kube-api-access-qccb2\") pod \"ingress-canary-fbvhc\" (UID: \"11bd6b74-116b-4233-a3f5-5b12c68be2f9\") " pod="openshift-ingress-canary/ingress-canary-fbvhc" Jul 16 21:48:07.964533 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:07.964416 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b50691f-e781-4af6-bb30-6600349a399f-metrics-tls\") pod \"dns-default-5hrrt\" (UID: \"7b50691f-e781-4af6-bb30-6600349a399f\") " pod="openshift-dns/dns-default-5hrrt" Jul 16 21:48:07.964533 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:07.964432 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7b50691f-e781-4af6-bb30-6600349a399f-config-volume\") pod \"dns-default-5hrrt\" (UID: \"7b50691f-e781-4af6-bb30-6600349a399f\") " pod="openshift-dns/dns-default-5hrrt" Jul 16 21:48:07.964533 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:07.964474 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr8ll\" (UniqueName: \"kubernetes.io/projected/7b50691f-e781-4af6-bb30-6600349a399f-kube-api-access-tr8ll\") pod \"dns-default-5hrrt\" (UID: \"7b50691f-e781-4af6-bb30-6600349a399f\") " pod="openshift-dns/dns-default-5hrrt" Jul 16 21:48:07.964533 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:07.964513 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 21:48:07.964753 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:07.964577 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b50691f-e781-4af6-bb30-6600349a399f-metrics-tls podName:7b50691f-e781-4af6-bb30-6600349a399f nodeName:}" failed. No retries permitted until 2026-07-16 21:48:08.464556963 +0000 UTC m=+37.086999913 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b50691f-e781-4af6-bb30-6600349a399f-metrics-tls") pod "dns-default-5hrrt" (UID: "7b50691f-e781-4af6-bb30-6600349a399f") : secret "dns-default-metrics-tls" not found Jul 16 21:48:07.964808 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:07.964799 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b50691f-e781-4af6-bb30-6600349a399f-tmp-dir\") pod \"dns-default-5hrrt\" (UID: \"7b50691f-e781-4af6-bb30-6600349a399f\") " pod="openshift-dns/dns-default-5hrrt" Jul 16 21:48:07.965030 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:07.965012 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7b50691f-e781-4af6-bb30-6600349a399f-config-volume\") pod \"dns-default-5hrrt\" (UID: \"7b50691f-e781-4af6-bb30-6600349a399f\") " pod="openshift-dns/dns-default-5hrrt" Jul 16 21:48:07.968595 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:07.968574 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11bd6b74-116b-4233-a3f5-5b12c68be2f9-cert\") pod \"ingress-canary-fbvhc\" (UID: \"11bd6b74-116b-4233-a3f5-5b12c68be2f9\") " pod="openshift-ingress-canary/ingress-canary-fbvhc" Jul 16 21:48:07.973553 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:07.973530 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qccb2\" (UniqueName: \"kubernetes.io/projected/11bd6b74-116b-4233-a3f5-5b12c68be2f9-kube-api-access-qccb2\") pod \"ingress-canary-fbvhc\" (UID: \"11bd6b74-116b-4233-a3f5-5b12c68be2f9\") " pod="openshift-ingress-canary/ingress-canary-fbvhc" Jul 16 21:48:07.974381 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:07.974364 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr8ll\" (UniqueName: \"kubernetes.io/projected/7b50691f-e781-4af6-bb30-6600349a399f-kube-api-access-tr8ll\") pod \"dns-default-5hrrt\" (UID: \"7b50691f-e781-4af6-bb30-6600349a399f\") " pod="openshift-dns/dns-default-5hrrt" Jul 16 21:48:08.117183 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:08.117154 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fbvhc" Jul 16 21:48:08.271835 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:08.271622 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fbvhc"] Jul 16 21:48:08.274813 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:48:08.274788 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11bd6b74_116b_4233_a3f5_5b12c68be2f9.slice/crio-c44ad27644954f360048fcb9c3ac014d5dd6233c7e88aaed70dd78d0ace8ebd5 WatchSource:0}: Error finding container c44ad27644954f360048fcb9c3ac014d5dd6233c7e88aaed70dd78d0ace8ebd5: Status 404 returned error can't find the container with id c44ad27644954f360048fcb9c3ac014d5dd6233c7e88aaed70dd78d0ace8ebd5 Jul 16 21:48:08.379404 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:08.379366 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:08.451791 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:08.451770 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:08.468882 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:08.468863 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b50691f-e781-4af6-bb30-6600349a399f-metrics-tls\") pod \"dns-default-5hrrt\" (UID: \"7b50691f-e781-4af6-bb30-6600349a399f\") " pod="openshift-dns/dns-default-5hrrt" Jul 16 21:48:08.468981 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:08.468968 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 21:48:08.469032 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:08.469023 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b50691f-e781-4af6-bb30-6600349a399f-metrics-tls podName:7b50691f-e781-4af6-bb30-6600349a399f nodeName:}" failed. No retries permitted until 2026-07-16 21:48:09.469010846 +0000 UTC m=+38.091453778 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b50691f-e781-4af6-bb30-6600349a399f-metrics-tls") pod "dns-default-5hrrt" (UID: "7b50691f-e781-4af6-bb30-6600349a399f") : secret "dns-default-metrics-tls" not found Jul 16 21:48:08.634121 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:08.634084 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7j8zw"] Jul 16 21:48:08.636364 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:08.636349 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7j8zw" Jul 16 21:48:08.647709 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:08.647691 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 16 21:48:08.647790 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:08.647731 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 16 21:48:08.647851 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:08.647818 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 16 21:48:08.647851 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:08.647818 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 16 21:48:08.648260 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:08.648248 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bcp4l\"" Jul 16 21:48:08.669570 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:08.669548 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/41c76e5f-e9ad-4b6b-b8c8-529fa032c570-data-volume\") pod \"insights-runtime-extractor-7j8zw\" (UID: \"41c76e5f-e9ad-4b6b-b8c8-529fa032c570\") " pod="openshift-insights/insights-runtime-extractor-7j8zw" Jul 16 21:48:08.669657 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:08.669599 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjqps\" (UniqueName: \"kubernetes.io/projected/41c76e5f-e9ad-4b6b-b8c8-529fa032c570-kube-api-access-tjqps\") pod \"insights-runtime-extractor-7j8zw\" (UID: \"41c76e5f-e9ad-4b6b-b8c8-529fa032c570\") " pod="openshift-insights/insights-runtime-extractor-7j8zw" Jul 16 21:48:08.669657 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:08.669638 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/41c76e5f-e9ad-4b6b-b8c8-529fa032c570-crio-socket\") pod \"insights-runtime-extractor-7j8zw\" (UID: \"41c76e5f-e9ad-4b6b-b8c8-529fa032c570\") " pod="openshift-insights/insights-runtime-extractor-7j8zw" Jul 16 21:48:08.669756 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:08.669660 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/41c76e5f-e9ad-4b6b-b8c8-529fa032c570-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7j8zw\" (UID: \"41c76e5f-e9ad-4b6b-b8c8-529fa032c570\") " pod="openshift-insights/insights-runtime-extractor-7j8zw" Jul 16 21:48:08.669756 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:08.669706 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/41c76e5f-e9ad-4b6b-b8c8-529fa032c570-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7j8zw\" (UID: \"41c76e5f-e9ad-4b6b-b8c8-529fa032c570\") " pod="openshift-insights/insights-runtime-extractor-7j8zw" Jul 16 21:48:08.670129 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:08.670111 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7j8zw"] Jul 16 21:48:08.770904 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:08.770876 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/41c76e5f-e9ad-4b6b-b8c8-529fa032c570-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7j8zw\" (UID: \"41c76e5f-e9ad-4b6b-b8c8-529fa032c570\") " pod="openshift-insights/insights-runtime-extractor-7j8zw" Jul 16 21:48:08.771046 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:08.770932 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/41c76e5f-e9ad-4b6b-b8c8-529fa032c570-data-volume\") pod \"insights-runtime-extractor-7j8zw\" (UID: \"41c76e5f-e9ad-4b6b-b8c8-529fa032c570\") " pod="openshift-insights/insights-runtime-extractor-7j8zw" Jul 16 21:48:08.771046 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:08.770983 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjqps\" (UniqueName: \"kubernetes.io/projected/41c76e5f-e9ad-4b6b-b8c8-529fa032c570-kube-api-access-tjqps\") pod \"insights-runtime-extractor-7j8zw\" (UID: \"41c76e5f-e9ad-4b6b-b8c8-529fa032c570\") " pod="openshift-insights/insights-runtime-extractor-7j8zw" Jul 16 21:48:08.771046 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:08.771030 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/41c76e5f-e9ad-4b6b-b8c8-529fa032c570-crio-socket\") pod \"insights-runtime-extractor-7j8zw\" (UID: \"41c76e5f-e9ad-4b6b-b8c8-529fa032c570\") " pod="openshift-insights/insights-runtime-extractor-7j8zw" Jul 16 21:48:08.771046 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:08.771036 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 21:48:08.771255 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:08.771057 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/41c76e5f-e9ad-4b6b-b8c8-529fa032c570-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7j8zw\" (UID: \"41c76e5f-e9ad-4b6b-b8c8-529fa032c570\") " pod="openshift-insights/insights-runtime-extractor-7j8zw" Jul 16 21:48:08.771255 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:08.771100 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41c76e5f-e9ad-4b6b-b8c8-529fa032c570-insights-runtime-extractor-tls podName:41c76e5f-e9ad-4b6b-b8c8-529fa032c570 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:09.271081804 +0000 UTC m=+37.893524762 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/41c76e5f-e9ad-4b6b-b8c8-529fa032c570-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7j8zw" (UID: "41c76e5f-e9ad-4b6b-b8c8-529fa032c570") : secret "insights-runtime-extractor-tls" not found Jul 16 21:48:08.771255 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:08.771142 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 21:48:08.771255 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:08.771245 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/41c76e5f-e9ad-4b6b-b8c8-529fa032c570-kube-rbac-proxy-cm podName:41c76e5f-e9ad-4b6b-b8c8-529fa032c570 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:09.27118355 +0000 UTC m=+37.893626511 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/41c76e5f-e9ad-4b6b-b8c8-529fa032c570-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7j8zw" (UID: "41c76e5f-e9ad-4b6b-b8c8-529fa032c570") : configmap "kube-rbac-proxy" not found Jul 16 21:48:08.771450 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:08.771317 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/41c76e5f-e9ad-4b6b-b8c8-529fa032c570-data-volume\") pod \"insights-runtime-extractor-7j8zw\" (UID: \"41c76e5f-e9ad-4b6b-b8c8-529fa032c570\") " pod="openshift-insights/insights-runtime-extractor-7j8zw" Jul 16 21:48:08.771450 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:08.771425 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/41c76e5f-e9ad-4b6b-b8c8-529fa032c570-crio-socket\") pod \"insights-runtime-extractor-7j8zw\" (UID: \"41c76e5f-e9ad-4b6b-b8c8-529fa032c570\") " pod="openshift-insights/insights-runtime-extractor-7j8zw" Jul 16 21:48:08.788438 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:08.788417 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjqps\" (UniqueName: \"kubernetes.io/projected/41c76e5f-e9ad-4b6b-b8c8-529fa032c570-kube-api-access-tjqps\") pod \"insights-runtime-extractor-7j8zw\" (UID: \"41c76e5f-e9ad-4b6b-b8c8-529fa032c570\") " pod="openshift-insights/insights-runtime-extractor-7j8zw" Jul 16 21:48:08.790473 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:08.790449 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-8d2s6"] Jul 16 21:48:08.792444 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:08.792429 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-8d2s6" Jul 16 21:48:08.797181 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:08.797154 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 16 21:48:08.798579 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:08.798561 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 16 21:48:08.798665 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:08.798647 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-h7q42\"" Jul 16 21:48:08.804652 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:08.804616 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:08.806055 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:08.806034 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-8d2s6"] Jul 16 21:48:08.871556 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:08.871532 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnzk6\" (UniqueName: \"kubernetes.io/projected/d5fb667b-f494-4f9f-9766-2d99f97b415b-kube-api-access-nnzk6\") pod \"migrator-7cc9c7ffd7-8d2s6\" (UID: \"d5fb667b-f494-4f9f-9766-2d99f97b415b\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-8d2s6" Jul 16 21:48:08.894972 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:08.894882 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:08.928766 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:08.928744 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwlrc" Jul 16 21:48:08.928766 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:08.928759 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvq8m" Jul 16 21:48:08.928916 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:08.928835 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcl48" Jul 16 21:48:08.931627 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:08.931587 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 21:48:08.931714 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:08.931672 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mc6l9\"" Jul 16 21:48:08.931767 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:08.931714 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 21:48:08.931767 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:08.931750 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 21:48:08.931868 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:08.931771 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6g595\"" Jul 16 21:48:08.931868 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:08.931812 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 21:48:08.972165 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:08.972143 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnzk6\" (UniqueName: \"kubernetes.io/projected/d5fb667b-f494-4f9f-9766-2d99f97b415b-kube-api-access-nnzk6\") pod \"migrator-7cc9c7ffd7-8d2s6\" (UID: \"d5fb667b-f494-4f9f-9766-2d99f97b415b\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-8d2s6" Jul 16 21:48:08.980829 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:08.980814 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnzk6\" (UniqueName: \"kubernetes.io/projected/d5fb667b-f494-4f9f-9766-2d99f97b415b-kube-api-access-nnzk6\") pod \"migrator-7cc9c7ffd7-8d2s6\" (UID: \"d5fb667b-f494-4f9f-9766-2d99f97b415b\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-8d2s6" Jul 16 21:48:09.088680 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:09.088647 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fbvhc" event={"ID":"11bd6b74-116b-4233-a3f5-5b12c68be2f9","Type":"ContainerStarted","Data":"c44ad27644954f360048fcb9c3ac014d5dd6233c7e88aaed70dd78d0ace8ebd5"} Jul 16 21:48:09.102300 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:09.102270 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:09.104119 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:09.104105 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-8d2s6" Jul 16 21:48:09.188396 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:09.188339 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:09.242059 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:09.242032 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-8d2s6"] Jul 16 21:48:09.245522 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:48:09.245495 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5fb667b_f494_4f9f_9766_2d99f97b415b.slice/crio-4c8c42446dfcb7423c602f69f74206a817b9e0428ba8c2e71d9987fa0d370d89 WatchSource:0}: Error finding container 4c8c42446dfcb7423c602f69f74206a817b9e0428ba8c2e71d9987fa0d370d89: Status 404 returned error can't find the container with id 4c8c42446dfcb7423c602f69f74206a817b9e0428ba8c2e71d9987fa0d370d89 Jul 16 21:48:09.275457 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:09.275431 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/41c76e5f-e9ad-4b6b-b8c8-529fa032c570-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7j8zw\" (UID: \"41c76e5f-e9ad-4b6b-b8c8-529fa032c570\") " pod="openshift-insights/insights-runtime-extractor-7j8zw" Jul 16 21:48:09.275561 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:09.275510 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/41c76e5f-e9ad-4b6b-b8c8-529fa032c570-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7j8zw\" (UID: \"41c76e5f-e9ad-4b6b-b8c8-529fa032c570\") " pod="openshift-insights/insights-runtime-extractor-7j8zw" Jul 16 21:48:09.275618 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:09.275588 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 21:48:09.275618 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:09.275602 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 21:48:09.275722 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:09.275649 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41c76e5f-e9ad-4b6b-b8c8-529fa032c570-insights-runtime-extractor-tls podName:41c76e5f-e9ad-4b6b-b8c8-529fa032c570 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:10.275627692 +0000 UTC m=+38.898070623 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/41c76e5f-e9ad-4b6b-b8c8-529fa032c570-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7j8zw" (UID: "41c76e5f-e9ad-4b6b-b8c8-529fa032c570") : secret "insights-runtime-extractor-tls" not found Jul 16 21:48:09.275722 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:09.275664 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/41c76e5f-e9ad-4b6b-b8c8-529fa032c570-kube-rbac-proxy-cm podName:41c76e5f-e9ad-4b6b-b8c8-529fa032c570 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:10.27565745 +0000 UTC m=+38.898100380 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/41c76e5f-e9ad-4b6b-b8c8-529fa032c570-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7j8zw" (UID: "41c76e5f-e9ad-4b6b-b8c8-529fa032c570") : configmap "kube-rbac-proxy" not found Jul 16 21:48:09.477709 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:09.477625 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b50691f-e781-4af6-bb30-6600349a399f-metrics-tls\") pod \"dns-default-5hrrt\" (UID: \"7b50691f-e781-4af6-bb30-6600349a399f\") " pod="openshift-dns/dns-default-5hrrt" Jul 16 21:48:09.477854 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:09.477806 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 21:48:09.477906 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:09.477872 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b50691f-e781-4af6-bb30-6600349a399f-metrics-tls podName:7b50691f-e781-4af6-bb30-6600349a399f nodeName:}" failed. No retries permitted until 2026-07-16 21:48:11.477857698 +0000 UTC m=+40.100300629 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b50691f-e781-4af6-bb30-6600349a399f-metrics-tls") pod "dns-default-5hrrt" (UID: "7b50691f-e781-4af6-bb30-6600349a399f") : secret "dns-default-metrics-tls" not found Jul 16 21:48:10.091814 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:10.091775 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-8d2s6" event={"ID":"d5fb667b-f494-4f9f-9766-2d99f97b415b","Type":"ContainerStarted","Data":"4c8c42446dfcb7423c602f69f74206a817b9e0428ba8c2e71d9987fa0d370d89"} Jul 16 21:48:10.284604 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:10.284570 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/41c76e5f-e9ad-4b6b-b8c8-529fa032c570-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7j8zw\" (UID: \"41c76e5f-e9ad-4b6b-b8c8-529fa032c570\") " pod="openshift-insights/insights-runtime-extractor-7j8zw" Jul 16 21:48:10.285043 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:10.284710 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/41c76e5f-e9ad-4b6b-b8c8-529fa032c570-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7j8zw\" (UID: \"41c76e5f-e9ad-4b6b-b8c8-529fa032c570\") " pod="openshift-insights/insights-runtime-extractor-7j8zw" Jul 16 21:48:10.285043 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:10.284742 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 21:48:10.285043 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:10.284783 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 21:48:10.285043 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:10.284811 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41c76e5f-e9ad-4b6b-b8c8-529fa032c570-insights-runtime-extractor-tls podName:41c76e5f-e9ad-4b6b-b8c8-529fa032c570 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:12.284793801 +0000 UTC m=+40.907236755 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/41c76e5f-e9ad-4b6b-b8c8-529fa032c570-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7j8zw" (UID: "41c76e5f-e9ad-4b6b-b8c8-529fa032c570") : secret "insights-runtime-extractor-tls" not found Jul 16 21:48:10.285043 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:10.284849 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/41c76e5f-e9ad-4b6b-b8c8-529fa032c570-kube-rbac-proxy-cm podName:41c76e5f-e9ad-4b6b-b8c8-529fa032c570 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:12.284820129 +0000 UTC m=+40.907263064 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/41c76e5f-e9ad-4b6b-b8c8-529fa032c570-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7j8zw" (UID: "41c76e5f-e9ad-4b6b-b8c8-529fa032c570") : configmap "kube-rbac-proxy" not found Jul 16 21:48:10.387431 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:10.387347 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7tltv_1dabb80a-8529-4289-82e4-e44ca61494c8/dns-node-resolver/0.log" Jul 16 21:48:10.685180 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:10.684883 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-t7sb4"] Jul 16 21:48:10.687378 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:10.687356 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-t7sb4" Jul 16 21:48:10.691398 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:10.691357 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 16 21:48:10.691552 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:10.691358 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 16 21:48:10.692408 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:10.692389 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-tgsrc\"" Jul 16 21:48:10.692493 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:10.692434 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 16 21:48:10.692493 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:10.692473 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 16 21:48:10.700420 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:10.700398 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-t7sb4"] Jul 16 21:48:10.730391 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:10.730362 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:10.788008 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:10.787974 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7585204f-0662-4588-b643-810311a09cbb-signing-key\") pod \"service-ca-58598695f7-t7sb4\" (UID: \"7585204f-0662-4588-b643-810311a09cbb\") " pod="openshift-service-ca/service-ca-58598695f7-t7sb4" Jul 16 21:48:10.788147 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:10.788019 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7585204f-0662-4588-b643-810311a09cbb-signing-cabundle\") pod \"service-ca-58598695f7-t7sb4\" (UID: \"7585204f-0662-4588-b643-810311a09cbb\") " pod="openshift-service-ca/service-ca-58598695f7-t7sb4" Jul 16 21:48:10.788147 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:10.788088 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n94pc\" (UniqueName: \"kubernetes.io/projected/7585204f-0662-4588-b643-810311a09cbb-kube-api-access-n94pc\") pod \"service-ca-58598695f7-t7sb4\" (UID: \"7585204f-0662-4588-b643-810311a09cbb\") " pod="openshift-service-ca/service-ca-58598695f7-t7sb4" Jul 16 21:48:10.819562 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:10.819529 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:10.889340 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:10.889307 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7585204f-0662-4588-b643-810311a09cbb-signing-key\") pod \"service-ca-58598695f7-t7sb4\" (UID: \"7585204f-0662-4588-b643-810311a09cbb\") " pod="openshift-service-ca/service-ca-58598695f7-t7sb4" Jul 16 21:48:10.889521 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:10.889347 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7585204f-0662-4588-b643-810311a09cbb-signing-cabundle\") pod \"service-ca-58598695f7-t7sb4\" (UID: \"7585204f-0662-4588-b643-810311a09cbb\") " pod="openshift-service-ca/service-ca-58598695f7-t7sb4" Jul 16 21:48:10.889521 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:10.889408 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n94pc\" (UniqueName: \"kubernetes.io/projected/7585204f-0662-4588-b643-810311a09cbb-kube-api-access-n94pc\") pod \"service-ca-58598695f7-t7sb4\" (UID: \"7585204f-0662-4588-b643-810311a09cbb\") " pod="openshift-service-ca/service-ca-58598695f7-t7sb4" Jul 16 21:48:10.890605 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:10.890577 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7585204f-0662-4588-b643-810311a09cbb-signing-cabundle\") pod \"service-ca-58598695f7-t7sb4\" (UID: \"7585204f-0662-4588-b643-810311a09cbb\") " pod="openshift-service-ca/service-ca-58598695f7-t7sb4" Jul 16 21:48:10.893151 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:10.893131 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7585204f-0662-4588-b643-810311a09cbb-signing-key\") pod \"service-ca-58598695f7-t7sb4\" (UID: \"7585204f-0662-4588-b643-810311a09cbb\") " pod="openshift-service-ca/service-ca-58598695f7-t7sb4" Jul 16 21:48:10.898200 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:10.898176 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n94pc\" (UniqueName: \"kubernetes.io/projected/7585204f-0662-4588-b643-810311a09cbb-kube-api-access-n94pc\") pod \"service-ca-58598695f7-t7sb4\" (UID: \"7585204f-0662-4588-b643-810311a09cbb\") " pod="openshift-service-ca/service-ca-58598695f7-t7sb4" Jul 16 21:48:11.000372 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:11.000300 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-t7sb4" Jul 16 21:48:11.096332 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:11.096297 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fbvhc" event={"ID":"11bd6b74-116b-4233-a3f5-5b12c68be2f9","Type":"ContainerStarted","Data":"45e4eaefbf4ef70942a9e10d3556e57f3b157408261f61ceda33cf0ca4f2c5da"} Jul 16 21:48:11.118065 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:11.118015 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fbvhc" podStartSLOduration=1.943826373 podStartE2EDuration="4.11799901s" totalImagesPullingTime="2.174172637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:48:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:11.117126952 +0000 UTC m=+39.739569907" watchObservedRunningTime="2026-07-16 21:48:11.11799901 +0000 UTC m=+39.740441968" Jul 16 21:48:11.252517 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:11.251646 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-t7sb4"] Jul 16 21:48:11.262158 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:48:11.262136 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7585204f_0662_4588_b643_810311a09cbb.slice/crio-01901f2011aac4931c8b507406d5d3789fb9916a9ca559553965301744ebf56a WatchSource:0}: Error finding container 01901f2011aac4931c8b507406d5d3789fb9916a9ca559553965301744ebf56a: Status 404 returned error can't find the container with id 01901f2011aac4931c8b507406d5d3789fb9916a9ca559553965301744ebf56a Jul 16 21:48:11.386433 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:11.386407 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-65cgd_f64092be-22df-4e1c-ab7e-4fcc7afb7ec7/node-ca/0.log" Jul 16 21:48:11.493990 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:11.493959 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b50691f-e781-4af6-bb30-6600349a399f-metrics-tls\") pod \"dns-default-5hrrt\" (UID: \"7b50691f-e781-4af6-bb30-6600349a399f\") " pod="openshift-dns/dns-default-5hrrt" Jul 16 21:48:11.494119 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:11.494107 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 21:48:11.494186 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:11.494174 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b50691f-e781-4af6-bb30-6600349a399f-metrics-tls podName:7b50691f-e781-4af6-bb30-6600349a399f nodeName:}" failed. No retries permitted until 2026-07-16 21:48:15.494154847 +0000 UTC m=+44.116597795 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b50691f-e781-4af6-bb30-6600349a399f-metrics-tls") pod "dns-default-5hrrt" (UID: "7b50691f-e781-4af6-bb30-6600349a399f") : secret "dns-default-metrics-tls" not found Jul 16 21:48:12.101116 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:12.101074 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-8d2s6" event={"ID":"d5fb667b-f494-4f9f-9766-2d99f97b415b","Type":"ContainerStarted","Data":"b707d0aa0565029348f392fddcd95944647412e6937afa001f28f474d2c8281f"} Jul 16 21:48:12.101314 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:12.101148 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-8d2s6" event={"ID":"d5fb667b-f494-4f9f-9766-2d99f97b415b","Type":"ContainerStarted","Data":"a0751a1d40a6f15ab39b50dbc56c1b438cbe6f9c632960859340614414101cf5"} Jul 16 21:48:12.102336 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:12.102309 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-t7sb4" event={"ID":"7585204f-0662-4588-b643-810311a09cbb","Type":"ContainerStarted","Data":"01901f2011aac4931c8b507406d5d3789fb9916a9ca559553965301744ebf56a"} Jul 16 21:48:12.125603 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:12.125541 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-8d2s6" podStartSLOduration=2.2247459689999998 podStartE2EDuration="4.125524074s" totalImagesPullingTime="1.900778105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:48:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:12.12470645 +0000 UTC m=+40.747149404" watchObservedRunningTime="2026-07-16 21:48:12.125524074 +0000 UTC m=+40.747967030" Jul 16 21:48:12.179660 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:12.179630 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:12.186291 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:12.186268 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fbvhc_11bd6b74-116b-4233-a3f5-5b12c68be2f9/serve-healthcheck-canary/0.log" Jul 16 21:48:12.299530 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:12.299493 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/41c76e5f-e9ad-4b6b-b8c8-529fa032c570-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7j8zw\" (UID: \"41c76e5f-e9ad-4b6b-b8c8-529fa032c570\") " pod="openshift-insights/insights-runtime-extractor-7j8zw" Jul 16 21:48:12.299699 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:12.299546 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/41c76e5f-e9ad-4b6b-b8c8-529fa032c570-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7j8zw\" (UID: \"41c76e5f-e9ad-4b6b-b8c8-529fa032c570\") " pod="openshift-insights/insights-runtime-extractor-7j8zw" Jul 16 21:48:12.299699 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:12.299631 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 21:48:12.299699 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:12.299647 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 21:48:12.299846 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:12.299709 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41c76e5f-e9ad-4b6b-b8c8-529fa032c570-insights-runtime-extractor-tls podName:41c76e5f-e9ad-4b6b-b8c8-529fa032c570 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:16.29969134 +0000 UTC m=+44.922134294 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/41c76e5f-e9ad-4b6b-b8c8-529fa032c570-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7j8zw" (UID: "41c76e5f-e9ad-4b6b-b8c8-529fa032c570") : secret "insights-runtime-extractor-tls" not found Jul 16 21:48:12.299846 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:12.299730 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/41c76e5f-e9ad-4b6b-b8c8-529fa032c570-kube-rbac-proxy-cm podName:41c76e5f-e9ad-4b6b-b8c8-529fa032c570 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:16.299719684 +0000 UTC m=+44.922162629 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/41c76e5f-e9ad-4b6b-b8c8-529fa032c570-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7j8zw" (UID: "41c76e5f-e9ad-4b6b-b8c8-529fa032c570") : configmap "kube-rbac-proxy" not found Jul 16 21:48:12.411162 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:12.411086 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:12.518195 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:12.518165 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:12.885561 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:12.885528 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:14.108972 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:14.108943 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-t7sb4" event={"ID":"7585204f-0662-4588-b643-810311a09cbb","Type":"ContainerStarted","Data":"132cd321db845a947443bacd40611bdc743854ce9a8d611c5f0d25074e4e750c"} Jul 16 21:48:14.128043 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:14.127994 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-t7sb4" podStartSLOduration=1.788085212 podStartE2EDuration="4.127977299s" totalImagesPullingTime="2.339892087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:48:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:14.126364745 +0000 UTC m=+42.748807698" watchObservedRunningTime="2026-07-16 21:48:14.127977299 +0000 UTC m=+42.750420248" Jul 16 21:48:14.748917 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:14.748879 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:14.849828 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:14.849788 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:14.956892 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:14.956856 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:15.051838 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:15.051806 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:15.149892 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:15.149855 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:15.526803 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:15.526712 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b50691f-e781-4af6-bb30-6600349a399f-metrics-tls\") pod \"dns-default-5hrrt\" (UID: \"7b50691f-e781-4af6-bb30-6600349a399f\") " pod="openshift-dns/dns-default-5hrrt" Jul 16 21:48:15.526950 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:15.526862 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 21:48:15.526950 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:15.526939 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b50691f-e781-4af6-bb30-6600349a399f-metrics-tls podName:7b50691f-e781-4af6-bb30-6600349a399f nodeName:}" failed. No retries permitted until 2026-07-16 21:48:23.526923405 +0000 UTC m=+52.149366341 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b50691f-e781-4af6-bb30-6600349a399f-metrics-tls") pod "dns-default-5hrrt" (UID: "7b50691f-e781-4af6-bb30-6600349a399f") : secret "dns-default-metrics-tls" not found Jul 16 21:48:15.653620 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:15.653587 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:15.850214 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:15.850182 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:16.251271 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:16.251177 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:16.332944 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:16.332911 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/41c76e5f-e9ad-4b6b-b8c8-529fa032c570-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7j8zw\" (UID: \"41c76e5f-e9ad-4b6b-b8c8-529fa032c570\") " pod="openshift-insights/insights-runtime-extractor-7j8zw" Jul 16 21:48:16.333146 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:16.332970 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/41c76e5f-e9ad-4b6b-b8c8-529fa032c570-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7j8zw\" (UID: \"41c76e5f-e9ad-4b6b-b8c8-529fa032c570\") " pod="openshift-insights/insights-runtime-extractor-7j8zw" Jul 16 21:48:16.333146 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:16.333133 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 21:48:16.333263 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:16.333213 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41c76e5f-e9ad-4b6b-b8c8-529fa032c570-insights-runtime-extractor-tls podName:41c76e5f-e9ad-4b6b-b8c8-529fa032c570 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:24.333191747 +0000 UTC m=+52.955634677 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/41c76e5f-e9ad-4b6b-b8c8-529fa032c570-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7j8zw" (UID: "41c76e5f-e9ad-4b6b-b8c8-529fa032c570") : secret "insights-runtime-extractor-tls" not found Jul 16 21:48:16.333482 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:16.333463 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/41c76e5f-e9ad-4b6b-b8c8-529fa032c570-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7j8zw\" (UID: \"41c76e5f-e9ad-4b6b-b8c8-529fa032c570\") " pod="openshift-insights/insights-runtime-extractor-7j8zw" Jul 16 21:48:16.646948 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:16.646920 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:17.116903 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:17.116871 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:17.466934 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:17.466864 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:17.862965 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:17.862933 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:18.131111 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:18.131029 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:18.229435 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:18.229408 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:18.527662 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:18.527593 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:18.645662 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:18.645637 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:18.908028 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:18.907999 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:19.053726 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:19.053698 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:19.556629 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:19.556584 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:19.652650 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:19.652610 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:19.877104 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:19.877018 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:20.305233 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:20.305191 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:20.470449 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:20.470414 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:20.695757 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:20.695683 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:20.858991 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:20.858958 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:21.272800 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:21.272767 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:21.863814 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:21.863782 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:22.298743 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:22.298704 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:22.492683 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:22.492648 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:22.882074 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:22.882043 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:22.978589 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:22.978526 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:23.588850 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:23.588817 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b50691f-e781-4af6-bb30-6600349a399f-metrics-tls\") pod \"dns-default-5hrrt\" (UID: \"7b50691f-e781-4af6-bb30-6600349a399f\") " pod="openshift-dns/dns-default-5hrrt" Jul 16 21:48:23.591951 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:23.591925 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b50691f-e781-4af6-bb30-6600349a399f-metrics-tls\") pod \"dns-default-5hrrt\" (UID: \"7b50691f-e781-4af6-bb30-6600349a399f\") " pod="openshift-dns/dns-default-5hrrt" Jul 16 21:48:23.711255 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:23.711208 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5hrrt" Jul 16 21:48:23.740739 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:23.740692 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:23.849190 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:23.849169 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5hrrt"] Jul 16 21:48:24.072289 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:24.072250 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:24.136668 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:24.136601 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5hrrt" event={"ID":"7b50691f-e781-4af6-bb30-6600349a399f","Type":"ContainerStarted","Data":"359058e102d617f563c71bfac2d6169e52242d5bc67b53e4c335de0043a1296f"} Jul 16 21:48:24.283999 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:24.283968 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:24.395531 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:24.395448 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/41c76e5f-e9ad-4b6b-b8c8-529fa032c570-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7j8zw\" (UID: \"41c76e5f-e9ad-4b6b-b8c8-529fa032c570\") " pod="openshift-insights/insights-runtime-extractor-7j8zw" Jul 16 21:48:24.398052 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:24.398025 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/41c76e5f-e9ad-4b6b-b8c8-529fa032c570-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7j8zw\" (UID: \"41c76e5f-e9ad-4b6b-b8c8-529fa032c570\") " pod="openshift-insights/insights-runtime-extractor-7j8zw" Jul 16 21:48:24.473874 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:24.473835 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:24.545735 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:24.545701 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7j8zw" Jul 16 21:48:24.713320 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:24.713293 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7j8zw"] Jul 16 21:48:24.715052 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:48:24.715021 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41c76e5f_e9ad_4b6b_b8c8_529fa032c570.slice/crio-9978d581b0b39ad5b25e46736ff5e40ad5f8d7c5721824d0cc02b0157270853a WatchSource:0}: Error finding container 9978d581b0b39ad5b25e46736ff5e40ad5f8d7c5721824d0cc02b0157270853a: Status 404 returned error can't find the container with id 9978d581b0b39ad5b25e46736ff5e40ad5f8d7c5721824d0cc02b0157270853a Jul 16 21:48:24.902231 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:24.902192 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:25.140097 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:25.140065 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7j8zw" event={"ID":"41c76e5f-e9ad-4b6b-b8c8-529fa032c570","Type":"ContainerStarted","Data":"9978d581b0b39ad5b25e46736ff5e40ad5f8d7c5721824d0cc02b0157270853a"} Jul 16 21:48:25.314813 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:25.314762 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:25.479015 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:25.478926 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:27.148650 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:27.148606 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5hrrt" event={"ID":"7b50691f-e781-4af6-bb30-6600349a399f","Type":"ContainerStarted","Data":"e221806786335d919f0df6af4c9a38278c28d0a3850ee9080a8acada62028352"} Jul 16 21:48:27.149032 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:27.148692 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5hrrt" event={"ID":"7b50691f-e781-4af6-bb30-6600349a399f","Type":"ContainerStarted","Data":"af092d535269296ab6cf13dcda6d15b938dcecaf36bb4fac97e2a45e63500250"} Jul 16 21:48:27.149032 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:27.148737 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5hrrt" Jul 16 21:48:27.176263 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:27.176190 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-5hrrt" podStartSLOduration=17.532708235 podStartE2EDuration="20.17617s" totalImagesPullingTime="2.643461765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:48:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:27.175768316 +0000 UTC m=+55.798211270" watchObservedRunningTime="2026-07-16 21:48:27.17617 +0000 UTC m=+55.798612954" Jul 16 21:48:30.159677 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:30.159643 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7j8zw" event={"ID":"41c76e5f-e9ad-4b6b-b8c8-529fa032c570","Type":"ContainerStarted","Data":"ddf42f74ae268db6c9b7f70819d88b2fed6b78608ec86acc36f8ffd93e191eb1"} Jul 16 21:48:30.667940 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:30.667910 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:30.778897 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:30.778869 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:31.130028 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:31.129986 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:31.251836 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:31.251750 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:31.348805 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:31.348767 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:32.416189 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.416152 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6867ffcdc4-6bknr"] Jul 16 21:48:32.423267 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.423242 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6867ffcdc4-6bknr" Jul 16 21:48:32.426832 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.426803 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 16 21:48:32.426947 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.426932 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 16 21:48:32.427480 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.427460 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-n26vv\"" Jul 16 21:48:32.427784 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.427767 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 16 21:48:32.428044 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.428029 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 16 21:48:32.437182 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.437159 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6867ffcdc4-6bknr"] Jul 16 21:48:32.437430 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.437364 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:32.528297 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.528267 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-njjsn"] Jul 16 21:48:32.531434 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.531411 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-njjsn" Jul 16 21:48:32.535130 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.535108 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 16 21:48:32.535627 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.535467 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 16 21:48:32.537453 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.537430 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-9xnj2\"" Jul 16 21:48:32.566562 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.566539 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-njjsn"] Jul 16 21:48:32.566731 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.566708 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w7qz\" (UniqueName: \"kubernetes.io/projected/0fa9f79c-4b7a-419f-9a31-f4f6135b9036-kube-api-access-5w7qz\") pod \"managed-serviceaccount-addon-agent-6867ffcdc4-6bknr\" (UID: \"0fa9f79c-4b7a-419f-9a31-f4f6135b9036\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6867ffcdc4-6bknr" Jul 16 21:48:32.566865 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.566849 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0fa9f79c-4b7a-419f-9a31-f4f6135b9036-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6867ffcdc4-6bknr\" (UID: \"0fa9f79c-4b7a-419f-9a31-f4f6135b9036\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6867ffcdc4-6bknr" Jul 16 21:48:32.594030 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.593998 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:32.606963 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.605620 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-55599968c8-d5h5x"] Jul 16 21:48:32.613589 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.612788 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55599968c8-d5h5x" Jul 16 21:48:32.617318 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.617290 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 16 21:48:32.617965 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.617941 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 16 21:48:32.618077 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.618015 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 16 21:48:32.624302 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.624280 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kdfjl\"" Jul 16 21:48:32.628776 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.628751 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55599968c8-d5h5x"] Jul 16 21:48:32.635247 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.635204 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 16 21:48:32.667794 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.667704 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w7qz\" (UniqueName: \"kubernetes.io/projected/0fa9f79c-4b7a-419f-9a31-f4f6135b9036-kube-api-access-5w7qz\") pod \"managed-serviceaccount-addon-agent-6867ffcdc4-6bknr\" (UID: \"0fa9f79c-4b7a-419f-9a31-f4f6135b9036\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6867ffcdc4-6bknr" Jul 16 21:48:32.667794 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.667787 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pkvn\" (UniqueName: \"kubernetes.io/projected/68e6c5e2-0a59-422d-a1c8-9b735597ed43-kube-api-access-7pkvn\") pod \"downloads-5bc67dcd46-njjsn\" (UID: \"68e6c5e2-0a59-422d-a1c8-9b735597ed43\") " pod="openshift-console/downloads-5bc67dcd46-njjsn" Jul 16 21:48:32.668007 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.667839 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0fa9f79c-4b7a-419f-9a31-f4f6135b9036-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6867ffcdc4-6bknr\" (UID: \"0fa9f79c-4b7a-419f-9a31-f4f6135b9036\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6867ffcdc4-6bknr" Jul 16 21:48:32.670982 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.670954 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0fa9f79c-4b7a-419f-9a31-f4f6135b9036-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6867ffcdc4-6bknr\" (UID: \"0fa9f79c-4b7a-419f-9a31-f4f6135b9036\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6867ffcdc4-6bknr" Jul 16 21:48:32.686381 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.686345 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w7qz\" (UniqueName: \"kubernetes.io/projected/0fa9f79c-4b7a-419f-9a31-f4f6135b9036-kube-api-access-5w7qz\") pod \"managed-serviceaccount-addon-agent-6867ffcdc4-6bknr\" (UID: \"0fa9f79c-4b7a-419f-9a31-f4f6135b9036\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6867ffcdc4-6bknr" Jul 16 21:48:32.711887 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.711339 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:32.743555 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.743524 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6867ffcdc4-6bknr" Jul 16 21:48:32.769096 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.769059 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/827c8e52-c812-41e1-9a4b-fd7478baa117-ca-trust-extracted\") pod \"image-registry-55599968c8-d5h5x\" (UID: \"827c8e52-c812-41e1-9a4b-fd7478baa117\") " pod="openshift-image-registry/image-registry-55599968c8-d5h5x" Jul 16 21:48:32.769258 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.769105 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/827c8e52-c812-41e1-9a4b-fd7478baa117-installation-pull-secrets\") pod \"image-registry-55599968c8-d5h5x\" (UID: \"827c8e52-c812-41e1-9a4b-fd7478baa117\") " pod="openshift-image-registry/image-registry-55599968c8-d5h5x" Jul 16 21:48:32.769258 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.769158 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pkvn\" (UniqueName: \"kubernetes.io/projected/68e6c5e2-0a59-422d-a1c8-9b735597ed43-kube-api-access-7pkvn\") pod \"downloads-5bc67dcd46-njjsn\" (UID: \"68e6c5e2-0a59-422d-a1c8-9b735597ed43\") " pod="openshift-console/downloads-5bc67dcd46-njjsn" Jul 16 21:48:32.769258 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.769196 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7svxn\" (UniqueName: \"kubernetes.io/projected/827c8e52-c812-41e1-9a4b-fd7478baa117-kube-api-access-7svxn\") pod \"image-registry-55599968c8-d5h5x\" (UID: \"827c8e52-c812-41e1-9a4b-fd7478baa117\") " pod="openshift-image-registry/image-registry-55599968c8-d5h5x" Jul 16 21:48:32.769392 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.769292 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/827c8e52-c812-41e1-9a4b-fd7478baa117-bound-sa-token\") pod \"image-registry-55599968c8-d5h5x\" (UID: \"827c8e52-c812-41e1-9a4b-fd7478baa117\") " pod="openshift-image-registry/image-registry-55599968c8-d5h5x" Jul 16 21:48:32.769466 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.769447 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/827c8e52-c812-41e1-9a4b-fd7478baa117-registry-tls\") pod \"image-registry-55599968c8-d5h5x\" (UID: \"827c8e52-c812-41e1-9a4b-fd7478baa117\") " pod="openshift-image-registry/image-registry-55599968c8-d5h5x" Jul 16 21:48:32.769506 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.769488 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/827c8e52-c812-41e1-9a4b-fd7478baa117-image-registry-private-configuration\") pod \"image-registry-55599968c8-d5h5x\" (UID: \"827c8e52-c812-41e1-9a4b-fd7478baa117\") " pod="openshift-image-registry/image-registry-55599968c8-d5h5x" Jul 16 21:48:32.769539 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.769516 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/827c8e52-c812-41e1-9a4b-fd7478baa117-registry-certificates\") pod \"image-registry-55599968c8-d5h5x\" (UID: \"827c8e52-c812-41e1-9a4b-fd7478baa117\") " pod="openshift-image-registry/image-registry-55599968c8-d5h5x" Jul 16 21:48:32.769592 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.769547 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/827c8e52-c812-41e1-9a4b-fd7478baa117-trusted-ca\") pod \"image-registry-55599968c8-d5h5x\" (UID: \"827c8e52-c812-41e1-9a4b-fd7478baa117\") " pod="openshift-image-registry/image-registry-55599968c8-d5h5x" Jul 16 21:48:32.789146 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.789123 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pkvn\" (UniqueName: \"kubernetes.io/projected/68e6c5e2-0a59-422d-a1c8-9b735597ed43-kube-api-access-7pkvn\") pod \"downloads-5bc67dcd46-njjsn\" (UID: \"68e6c5e2-0a59-422d-a1c8-9b735597ed43\") " pod="openshift-console/downloads-5bc67dcd46-njjsn" Jul 16 21:48:32.834489 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.834460 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:32.842542 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.842516 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-njjsn" Jul 16 21:48:32.870719 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.870676 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/827c8e52-c812-41e1-9a4b-fd7478baa117-trusted-ca\") pod \"image-registry-55599968c8-d5h5x\" (UID: \"827c8e52-c812-41e1-9a4b-fd7478baa117\") " pod="openshift-image-registry/image-registry-55599968c8-d5h5x" Jul 16 21:48:32.870850 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.870759 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/827c8e52-c812-41e1-9a4b-fd7478baa117-ca-trust-extracted\") pod \"image-registry-55599968c8-d5h5x\" (UID: \"827c8e52-c812-41e1-9a4b-fd7478baa117\") " pod="openshift-image-registry/image-registry-55599968c8-d5h5x" Jul 16 21:48:32.870850 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.870784 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/827c8e52-c812-41e1-9a4b-fd7478baa117-installation-pull-secrets\") pod \"image-registry-55599968c8-d5h5x\" (UID: \"827c8e52-c812-41e1-9a4b-fd7478baa117\") " pod="openshift-image-registry/image-registry-55599968c8-d5h5x" Jul 16 21:48:32.870850 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.870836 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7svxn\" (UniqueName: \"kubernetes.io/projected/827c8e52-c812-41e1-9a4b-fd7478baa117-kube-api-access-7svxn\") pod \"image-registry-55599968c8-d5h5x\" (UID: \"827c8e52-c812-41e1-9a4b-fd7478baa117\") " pod="openshift-image-registry/image-registry-55599968c8-d5h5x" Jul 16 21:48:32.871005 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.870882 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/827c8e52-c812-41e1-9a4b-fd7478baa117-bound-sa-token\") pod \"image-registry-55599968c8-d5h5x\" (UID: \"827c8e52-c812-41e1-9a4b-fd7478baa117\") " pod="openshift-image-registry/image-registry-55599968c8-d5h5x" Jul 16 21:48:32.871005 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.870911 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/827c8e52-c812-41e1-9a4b-fd7478baa117-registry-tls\") pod \"image-registry-55599968c8-d5h5x\" (UID: \"827c8e52-c812-41e1-9a4b-fd7478baa117\") " pod="openshift-image-registry/image-registry-55599968c8-d5h5x" Jul 16 21:48:32.871005 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.870940 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/827c8e52-c812-41e1-9a4b-fd7478baa117-image-registry-private-configuration\") pod \"image-registry-55599968c8-d5h5x\" (UID: \"827c8e52-c812-41e1-9a4b-fd7478baa117\") " pod="openshift-image-registry/image-registry-55599968c8-d5h5x" Jul 16 21:48:32.871005 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.870965 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/827c8e52-c812-41e1-9a4b-fd7478baa117-registry-certificates\") pod \"image-registry-55599968c8-d5h5x\" (UID: \"827c8e52-c812-41e1-9a4b-fd7478baa117\") " pod="openshift-image-registry/image-registry-55599968c8-d5h5x" Jul 16 21:48:32.871812 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.871786 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/827c8e52-c812-41e1-9a4b-fd7478baa117-registry-certificates\") pod \"image-registry-55599968c8-d5h5x\" (UID: \"827c8e52-c812-41e1-9a4b-fd7478baa117\") " pod="openshift-image-registry/image-registry-55599968c8-d5h5x" Jul 16 21:48:32.872872 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.872612 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/827c8e52-c812-41e1-9a4b-fd7478baa117-ca-trust-extracted\") pod \"image-registry-55599968c8-d5h5x\" (UID: \"827c8e52-c812-41e1-9a4b-fd7478baa117\") " pod="openshift-image-registry/image-registry-55599968c8-d5h5x" Jul 16 21:48:32.872872 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.872672 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/827c8e52-c812-41e1-9a4b-fd7478baa117-trusted-ca\") pod \"image-registry-55599968c8-d5h5x\" (UID: \"827c8e52-c812-41e1-9a4b-fd7478baa117\") " pod="openshift-image-registry/image-registry-55599968c8-d5h5x" Jul 16 21:48:32.875235 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.875193 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/827c8e52-c812-41e1-9a4b-fd7478baa117-image-registry-private-configuration\") pod \"image-registry-55599968c8-d5h5x\" (UID: \"827c8e52-c812-41e1-9a4b-fd7478baa117\") " pod="openshift-image-registry/image-registry-55599968c8-d5h5x" Jul 16 21:48:32.875325 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.875242 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/827c8e52-c812-41e1-9a4b-fd7478baa117-registry-tls\") pod \"image-registry-55599968c8-d5h5x\" (UID: \"827c8e52-c812-41e1-9a4b-fd7478baa117\") " pod="openshift-image-registry/image-registry-55599968c8-d5h5x" Jul 16 21:48:32.875561 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.875533 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/827c8e52-c812-41e1-9a4b-fd7478baa117-installation-pull-secrets\") pod \"image-registry-55599968c8-d5h5x\" (UID: \"827c8e52-c812-41e1-9a4b-fd7478baa117\") " pod="openshift-image-registry/image-registry-55599968c8-d5h5x" Jul 16 21:48:32.893806 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.893779 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7svxn\" (UniqueName: \"kubernetes.io/projected/827c8e52-c812-41e1-9a4b-fd7478baa117-kube-api-access-7svxn\") pod \"image-registry-55599968c8-d5h5x\" (UID: \"827c8e52-c812-41e1-9a4b-fd7478baa117\") " pod="openshift-image-registry/image-registry-55599968c8-d5h5x" Jul 16 21:48:32.894770 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.894748 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/827c8e52-c812-41e1-9a4b-fd7478baa117-bound-sa-token\") pod \"image-registry-55599968c8-d5h5x\" (UID: \"827c8e52-c812-41e1-9a4b-fd7478baa117\") " pod="openshift-image-registry/image-registry-55599968c8-d5h5x" Jul 16 21:48:32.924826 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.924748 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55599968c8-d5h5x" Jul 16 21:48:32.974621 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:32.974589 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:33.101598 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:33.101564 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:33.207094 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:33.207002 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:33.343512 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:33.343481 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:34.087856 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:34.087655 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lmrcx" Jul 16 21:48:34.131052 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:34.130991 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:34.198546 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:34.198516 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55599968c8-d5h5x"] Jul 16 21:48:34.206859 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:34.206830 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-njjsn"] Jul 16 21:48:34.222588 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:34.222544 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6867ffcdc4-6bknr"] Jul 16 21:48:34.272622 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:34.272595 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:34.316649 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:48:34.316564 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod827c8e52_c812_41e1_9a4b_fd7478baa117.slice/crio-a425e880c0e3279aee3b6bbfc4e29a401cf2f272d5d0da42bd5ff201e4d4094f WatchSource:0}: Error finding container a425e880c0e3279aee3b6bbfc4e29a401cf2f272d5d0da42bd5ff201e4d4094f: Status 404 returned error can't find the container with id a425e880c0e3279aee3b6bbfc4e29a401cf2f272d5d0da42bd5ff201e4d4094f Jul 16 21:48:34.317840 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:48:34.317806 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68e6c5e2_0a59_422d_a1c8_9b735597ed43.slice/crio-8748b8c8794e5fcab5bdc829d070ef21cce3dff095635b7b94f5e07d717bc94e WatchSource:0}: Error finding container 8748b8c8794e5fcab5bdc829d070ef21cce3dff095635b7b94f5e07d717bc94e: Status 404 returned error can't find the container with id 8748b8c8794e5fcab5bdc829d070ef21cce3dff095635b7b94f5e07d717bc94e Jul 16 21:48:34.318158 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:48:34.318134 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0fa9f79c_4b7a_419f_9a31_f4f6135b9036.slice/crio-ee9ab867666fbcf0c176f4b7d7c525b2ef9012263374ec0bf8c8b875ce44a247 WatchSource:0}: Error finding container ee9ab867666fbcf0c176f4b7d7c525b2ef9012263374ec0bf8c8b875ce44a247: Status 404 returned error can't find the container with id ee9ab867666fbcf0c176f4b7d7c525b2ef9012263374ec0bf8c8b875ce44a247 Jul 16 21:48:34.373997 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:34.373975 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:34.478493 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:34.478467 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:35.180085 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:35.180043 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55599968c8-d5h5x" event={"ID":"827c8e52-c812-41e1-9a4b-fd7478baa117","Type":"ContainerStarted","Data":"1f11b148d927145067211c09e7c7576fe5d5f78dbecbb1184b11e7ce5becb469"} Jul 16 21:48:35.180535 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:35.180113 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55599968c8-d5h5x" event={"ID":"827c8e52-c812-41e1-9a4b-fd7478baa117","Type":"ContainerStarted","Data":"a425e880c0e3279aee3b6bbfc4e29a401cf2f272d5d0da42bd5ff201e4d4094f"} Jul 16 21:48:35.180860 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:35.180833 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-55599968c8-d5h5x" Jul 16 21:48:35.183278 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:35.183251 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7j8zw" event={"ID":"41c76e5f-e9ad-4b6b-b8c8-529fa032c570","Type":"ContainerStarted","Data":"67600d8a398358893700792442e666a17dc6f36eb5729352759fc1d6abf9373a"} Jul 16 21:48:35.185024 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:35.184977 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-njjsn" event={"ID":"68e6c5e2-0a59-422d-a1c8-9b735597ed43","Type":"ContainerStarted","Data":"8748b8c8794e5fcab5bdc829d070ef21cce3dff095635b7b94f5e07d717bc94e"} Jul 16 21:48:35.186463 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:35.186428 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6867ffcdc4-6bknr" event={"ID":"0fa9f79c-4b7a-419f-9a31-f4f6135b9036","Type":"ContainerStarted","Data":"ee9ab867666fbcf0c176f4b7d7c525b2ef9012263374ec0bf8c8b875ce44a247"} Jul 16 21:48:35.204556 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:35.204512 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-55599968c8-d5h5x" podStartSLOduration=3.204497919 podStartE2EDuration="3.204497919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:48:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:35.203677691 +0000 UTC m=+63.826120644" watchObservedRunningTime="2026-07-16 21:48:35.204497919 +0000 UTC m=+63.826940868" Jul 16 21:48:37.153845 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:37.153808 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5hrrt" Jul 16 21:48:37.715641 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:37.715605 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9b9235a-c4fc-4aac-93b4-2964356667f1-original-pull-secret\") pod \"global-pull-secret-syncer-jwlrc\" (UID: \"e9b9235a-c4fc-4aac-93b4-2964356667f1\") " pod="kube-system/global-pull-secret-syncer-jwlrc" Jul 16 21:48:37.715862 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:37.715717 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c33b5633-01c6-4f48-8cb7-6969a24fe86b-metrics-certs\") pod \"network-metrics-daemon-lcl48\" (UID: \"c33b5633-01c6-4f48-8cb7-6969a24fe86b\") " pod="openshift-multus/network-metrics-daemon-lcl48" Jul 16 21:48:37.718506 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:37.718483 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 21:48:37.718639 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:37.718484 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 21:48:37.729240 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:37.729164 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9b9235a-c4fc-4aac-93b4-2964356667f1-original-pull-secret\") pod \"global-pull-secret-syncer-jwlrc\" (UID: \"e9b9235a-c4fc-4aac-93b4-2964356667f1\") " pod="kube-system/global-pull-secret-syncer-jwlrc" Jul 16 21:48:37.729681 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:37.729660 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c33b5633-01c6-4f48-8cb7-6969a24fe86b-metrics-certs\") pod \"network-metrics-daemon-lcl48\" (UID: \"c33b5633-01c6-4f48-8cb7-6969a24fe86b\") " pod="openshift-multus/network-metrics-daemon-lcl48" Jul 16 21:48:37.739850 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:37.739825 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwlrc" Jul 16 21:48:37.750786 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:37.750761 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6g595\"" Jul 16 21:48:37.758001 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:37.757976 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcl48" Jul 16 21:48:37.816206 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:37.816110 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4hrg\" (UniqueName: \"kubernetes.io/projected/f79c076a-55a4-487c-a763-9a9923ba4129-kube-api-access-x4hrg\") pod \"network-check-target-zvq8m\" (UID: \"f79c076a-55a4-487c-a763-9a9923ba4129\") " pod="openshift-network-diagnostics/network-check-target-zvq8m" Jul 16 21:48:37.819263 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:37.819233 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 21:48:37.829650 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:37.829629 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 21:48:37.839950 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:37.839924 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4hrg\" (UniqueName: \"kubernetes.io/projected/f79c076a-55a4-487c-a763-9a9923ba4129-kube-api-access-x4hrg\") pod \"network-check-target-zvq8m\" (UID: \"f79c076a-55a4-487c-a763-9a9923ba4129\") " pod="openshift-network-diagnostics/network-check-target-zvq8m" Jul 16 21:48:38.055696 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:38.055662 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mc6l9\"" Jul 16 21:48:38.063167 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:38.063139 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvq8m" Jul 16 21:48:38.342884 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:38.342803 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:38.415936 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:38.415902 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:39.165940 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:39.165899 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:39.367414 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:39.367382 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:39.400994 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:39.400960 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-zcmjn"] Jul 16 21:48:39.418468 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:39.418390 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-zcmjn"] Jul 16 21:48:39.418622 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:39.418505 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-zcmjn" Jul 16 21:48:39.421457 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:39.421430 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 16 21:48:39.421800 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:39.421431 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-wdpf6\"" Jul 16 21:48:39.503960 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:39.503865 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:39.531267 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:39.531157 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/bc97620a-757d-4bcb-ab6c-6a021f675d55-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-zcmjn\" (UID: \"bc97620a-757d-4bcb-ab6c-6a021f675d55\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-zcmjn" Jul 16 21:48:39.630339 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:39.630304 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:39.632511 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:39.632345 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/bc97620a-757d-4bcb-ab6c-6a021f675d55-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-zcmjn\" (UID: \"bc97620a-757d-4bcb-ab6c-6a021f675d55\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-zcmjn" Jul 16 21:48:39.635370 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:39.635322 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/bc97620a-757d-4bcb-ab6c-6a021f675d55-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-zcmjn\" (UID: \"bc97620a-757d-4bcb-ab6c-6a021f675d55\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-zcmjn" Jul 16 21:48:39.731847 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:39.731451 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-zcmjn" Jul 16 21:48:39.734780 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:39.734702 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:39.840915 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:39.840880 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:39.865630 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:39.865601 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jwlrc"] Jul 16 21:48:39.866547 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:48:39.866508 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9b9235a_c4fc_4aac_93b4_2964356667f1.slice/crio-770594904f80a989acdc3a443bc68290551b7e74bf17e7469fe57782bc0aabf5 WatchSource:0}: Error finding container 770594904f80a989acdc3a443bc68290551b7e74bf17e7469fe57782bc0aabf5: Status 404 returned error can't find the container with id 770594904f80a989acdc3a443bc68290551b7e74bf17e7469fe57782bc0aabf5 Jul 16 21:48:39.866673 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:39.866594 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zvq8m"] Jul 16 21:48:39.884137 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:39.884073 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lcl48"] Jul 16 21:48:39.886166 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:48:39.886134 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc33b5633_01c6_4f48_8cb7_6969a24fe86b.slice/crio-17f4932980a6eb0c375d37d4ce008e1b054a71e26ebee3964e6efbb8b817d562 WatchSource:0}: Error finding container 17f4932980a6eb0c375d37d4ce008e1b054a71e26ebee3964e6efbb8b817d562: Status 404 returned error can't find the container with id 17f4932980a6eb0c375d37d4ce008e1b054a71e26ebee3964e6efbb8b817d562 Jul 16 21:48:39.944892 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:39.944822 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-zcmjn"] Jul 16 21:48:39.952808 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:48:39.952777 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc97620a_757d_4bcb_ab6c_6a021f675d55.slice/crio-523abd14c5e24d0c23fddbd71fc15bd116fd58513772a0d7d2d3947ab9c4e862 WatchSource:0}: Error finding container 523abd14c5e24d0c23fddbd71fc15bd116fd58513772a0d7d2d3947ab9c4e862: Status 404 returned error can't find the container with id 523abd14c5e24d0c23fddbd71fc15bd116fd58513772a0d7d2d3947ab9c4e862 Jul 16 21:48:40.212315 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:40.212228 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jwlrc" event={"ID":"e9b9235a-c4fc-4aac-93b4-2964356667f1","Type":"ContainerStarted","Data":"770594904f80a989acdc3a443bc68290551b7e74bf17e7469fe57782bc0aabf5"} Jul 16 21:48:40.213506 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:40.213480 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zvq8m" event={"ID":"f79c076a-55a4-487c-a763-9a9923ba4129","Type":"ContainerStarted","Data":"ae633702855bc33a580657bef95e099fce2f7c4bbb5e78cb634df41607355263"} Jul 16 21:48:40.214757 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:40.214732 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lcl48" event={"ID":"c33b5633-01c6-4f48-8cb7-6969a24fe86b","Type":"ContainerStarted","Data":"17f4932980a6eb0c375d37d4ce008e1b054a71e26ebee3964e6efbb8b817d562"} Jul 16 21:48:40.217027 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:40.216997 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7j8zw" event={"ID":"41c76e5f-e9ad-4b6b-b8c8-529fa032c570","Type":"ContainerStarted","Data":"b7811cca0ff92231140f0b8d989c74e9ff05e89e4b4fe947935d9e933153d3bc"} Jul 16 21:48:40.218669 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:40.218646 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6867ffcdc4-6bknr" event={"ID":"0fa9f79c-4b7a-419f-9a31-f4f6135b9036","Type":"ContainerStarted","Data":"1f3315532e72395cba088d65be46fd7ec404880904110968f8688bee56ea9a6a"} Jul 16 21:48:40.219786 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:40.219767 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-zcmjn" event={"ID":"bc97620a-757d-4bcb-ab6c-6a021f675d55","Type":"ContainerStarted","Data":"523abd14c5e24d0c23fddbd71fc15bd116fd58513772a0d7d2d3947ab9c4e862"} Jul 16 21:48:40.246946 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:40.246900 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7j8zw" podStartSLOduration=17.479917032 podStartE2EDuration="32.246885983s" totalImagesPullingTime="14.766968951s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:48:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:40.246538293 +0000 UTC m=+68.868981247" watchObservedRunningTime="2026-07-16 21:48:40.246885983 +0000 UTC m=+68.869328936" Jul 16 21:48:40.286118 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:40.286066 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6867ffcdc4-6bknr" podStartSLOduration=2.994420118 podStartE2EDuration="8.286050065s" totalImagesPullingTime="5.291629947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:40.283500949 +0000 UTC m=+68.905943901" watchObservedRunningTime="2026-07-16 21:48:40.286050065 +0000 UTC m=+68.908493017" Jul 16 21:48:40.391797 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:40.391532 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:40.527643 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:40.527557 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:42.233375 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:42.233331 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lcl48" event={"ID":"c33b5633-01c6-4f48-8cb7-6969a24fe86b","Type":"ContainerStarted","Data":"6e4de3a1390fb4c1d2ec8ac47d9bb0362684544318650d7366d9c8dcb96d9075"} Jul 16 21:48:42.233857 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:42.233472 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lcl48" event={"ID":"c33b5633-01c6-4f48-8cb7-6969a24fe86b","Type":"ContainerStarted","Data":"c5aa6b1423d6cf99c803e1b54a6359f782dbd9ede3506ad77e4dac1126128a56"} Jul 16 21:48:42.236084 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:42.236054 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-zcmjn" event={"ID":"bc97620a-757d-4bcb-ab6c-6a021f675d55","Type":"ContainerStarted","Data":"b38bbb4145e7e323805d0eae69bc1977eb8a9faf2ec13db18077f511a73979ec"} Jul 16 21:48:42.236312 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:42.236291 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-zcmjn" Jul 16 21:48:42.242857 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:42.242832 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-zcmjn" Jul 16 21:48:42.258123 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:42.257952 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lcl48" podStartSLOduration=68.446650693 podStartE2EDuration="1m10.25792963s" totalImagesPullingTime="1.811278937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:47:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:42.25682735 +0000 UTC m=+70.879270304" watchObservedRunningTime="2026-07-16 21:48:42.25792963 +0000 UTC m=+70.880372584" Jul 16 21:48:42.327082 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:42.327007 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:42.441154 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:42.441123 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:42.543037 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:42.542950 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:43.420905 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:43.420872 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:43.515427 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:43.515390 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:43.620260 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:43.620202 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:43.744407 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:43.744324 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:43.840676 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:43.840597 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:45.180081 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:45.179976 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:45.287192 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:45.287152 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:45.945526 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:45.945491 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:46.254831 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:46.254742 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jwlrc" event={"ID":"e9b9235a-c4fc-4aac-93b4-2964356667f1","Type":"ContainerStarted","Data":"a2e9cbd07855eb16f4296d325cfe570a533487f99b7dce0c1f43ef0e983da0da"} Jul 16 21:48:46.256462 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:46.256433 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zvq8m" event={"ID":"f79c076a-55a4-487c-a763-9a9923ba4129","Type":"ContainerStarted","Data":"c33ae824c1ebf91e1d462957462763ea88300a82730a4fa2faf659862af6684c"} Jul 16 21:48:46.256598 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:46.256587 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zvq8m" Jul 16 21:48:46.275943 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:46.275889 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-zcmjn" podStartSLOduration=5.535177876 podStartE2EDuration="7.275873303s" totalImagesPullingTime="1.740695427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:48:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:42.293838804 +0000 UTC m=+70.916281757" watchObservedRunningTime="2026-07-16 21:48:46.275873303 +0000 UTC m=+74.898316255" Jul 16 21:48:46.300117 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:46.300049 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jwlrc" podStartSLOduration=68.691591734 podStartE2EDuration="1m14.300024342s" totalImagesPullingTime="5.608432608s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:47:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:46.275968948 +0000 UTC m=+74.898411902" watchObservedRunningTime="2026-07-16 21:48:46.300024342 +0000 UTC m=+74.922467294" Jul 16 21:48:46.347361 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:46.347315 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:46.465854 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:46.465818 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:46.748115 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:46.748079 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:46.850578 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:46.850540 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:46.931076 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:46.931016 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zvq8m" podStartSLOduration=69.324751151 podStartE2EDuration="1m14.930998459s" totalImagesPullingTime="5.606247308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:47:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:46.302173198 +0000 UTC m=+74.924616150" watchObservedRunningTime="2026-07-16 21:48:46.930998459 +0000 UTC m=+75.553441411" Jul 16 21:48:46.931833 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:46.931813 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8d454b4cf-rwfcx"] Jul 16 21:48:46.934979 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:46.934961 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8d454b4cf-rwfcx" Jul 16 21:48:46.938956 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:46.938933 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fsgdm\"" Jul 16 21:48:46.939094 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:46.938964 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 16 21:48:46.939094 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:46.939014 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 16 21:48:46.939094 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:46.938937 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 16 21:48:46.939380 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:46.939358 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 16 21:48:46.939460 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:46.939358 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 16 21:48:46.962300 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:46.962076 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 16 21:48:46.962300 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:46.962085 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8d454b4cf-rwfcx"] Jul 16 21:48:47.097933 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:47.097897 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cxtn\" (UniqueName: \"kubernetes.io/projected/c3630f3d-ba54-42c2-8fa4-315214cf095a-kube-api-access-9cxtn\") pod \"console-8d454b4cf-rwfcx\" (UID: \"c3630f3d-ba54-42c2-8fa4-315214cf095a\") " pod="openshift-console/console-8d454b4cf-rwfcx" Jul 16 21:48:47.097933 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:47.097939 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3630f3d-ba54-42c2-8fa4-315214cf095a-trusted-ca-bundle\") pod \"console-8d454b4cf-rwfcx\" (UID: \"c3630f3d-ba54-42c2-8fa4-315214cf095a\") " pod="openshift-console/console-8d454b4cf-rwfcx" Jul 16 21:48:47.098199 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:47.097958 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c3630f3d-ba54-42c2-8fa4-315214cf095a-console-config\") pod \"console-8d454b4cf-rwfcx\" (UID: \"c3630f3d-ba54-42c2-8fa4-315214cf095a\") " pod="openshift-console/console-8d454b4cf-rwfcx" Jul 16 21:48:47.098199 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:47.098000 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c3630f3d-ba54-42c2-8fa4-315214cf095a-oauth-serving-cert\") pod \"console-8d454b4cf-rwfcx\" (UID: \"c3630f3d-ba54-42c2-8fa4-315214cf095a\") " pod="openshift-console/console-8d454b4cf-rwfcx" Jul 16 21:48:47.098199 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:47.098066 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c3630f3d-ba54-42c2-8fa4-315214cf095a-service-ca\") pod \"console-8d454b4cf-rwfcx\" (UID: \"c3630f3d-ba54-42c2-8fa4-315214cf095a\") " pod="openshift-console/console-8d454b4cf-rwfcx" Jul 16 21:48:47.098199 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:47.098110 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c3630f3d-ba54-42c2-8fa4-315214cf095a-console-oauth-config\") pod \"console-8d454b4cf-rwfcx\" (UID: \"c3630f3d-ba54-42c2-8fa4-315214cf095a\") " pod="openshift-console/console-8d454b4cf-rwfcx" Jul 16 21:48:47.098199 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:47.098148 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c3630f3d-ba54-42c2-8fa4-315214cf095a-console-serving-cert\") pod \"console-8d454b4cf-rwfcx\" (UID: \"c3630f3d-ba54-42c2-8fa4-315214cf095a\") " pod="openshift-console/console-8d454b4cf-rwfcx" Jul 16 21:48:47.199020 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:47.198982 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c3630f3d-ba54-42c2-8fa4-315214cf095a-oauth-serving-cert\") pod \"console-8d454b4cf-rwfcx\" (UID: \"c3630f3d-ba54-42c2-8fa4-315214cf095a\") " pod="openshift-console/console-8d454b4cf-rwfcx" Jul 16 21:48:47.199205 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:47.199032 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c3630f3d-ba54-42c2-8fa4-315214cf095a-service-ca\") pod \"console-8d454b4cf-rwfcx\" (UID: \"c3630f3d-ba54-42c2-8fa4-315214cf095a\") " pod="openshift-console/console-8d454b4cf-rwfcx" Jul 16 21:48:47.199205 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:47.199067 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c3630f3d-ba54-42c2-8fa4-315214cf095a-console-oauth-config\") pod \"console-8d454b4cf-rwfcx\" (UID: \"c3630f3d-ba54-42c2-8fa4-315214cf095a\") " pod="openshift-console/console-8d454b4cf-rwfcx" Jul 16 21:48:47.199205 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:47.199089 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c3630f3d-ba54-42c2-8fa4-315214cf095a-console-serving-cert\") pod \"console-8d454b4cf-rwfcx\" (UID: \"c3630f3d-ba54-42c2-8fa4-315214cf095a\") " pod="openshift-console/console-8d454b4cf-rwfcx" Jul 16 21:48:47.199205 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:47.199136 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cxtn\" (UniqueName: \"kubernetes.io/projected/c3630f3d-ba54-42c2-8fa4-315214cf095a-kube-api-access-9cxtn\") pod \"console-8d454b4cf-rwfcx\" (UID: \"c3630f3d-ba54-42c2-8fa4-315214cf095a\") " pod="openshift-console/console-8d454b4cf-rwfcx" Jul 16 21:48:47.199205 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:47.199155 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3630f3d-ba54-42c2-8fa4-315214cf095a-trusted-ca-bundle\") pod \"console-8d454b4cf-rwfcx\" (UID: \"c3630f3d-ba54-42c2-8fa4-315214cf095a\") " pod="openshift-console/console-8d454b4cf-rwfcx" Jul 16 21:48:47.199205 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:47.199174 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c3630f3d-ba54-42c2-8fa4-315214cf095a-console-config\") pod \"console-8d454b4cf-rwfcx\" (UID: \"c3630f3d-ba54-42c2-8fa4-315214cf095a\") " pod="openshift-console/console-8d454b4cf-rwfcx" Jul 16 21:48:47.199929 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:47.199896 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c3630f3d-ba54-42c2-8fa4-315214cf095a-service-ca\") pod \"console-8d454b4cf-rwfcx\" (UID: \"c3630f3d-ba54-42c2-8fa4-315214cf095a\") " pod="openshift-console/console-8d454b4cf-rwfcx" Jul 16 21:48:47.200029 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:47.199903 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c3630f3d-ba54-42c2-8fa4-315214cf095a-oauth-serving-cert\") pod \"console-8d454b4cf-rwfcx\" (UID: \"c3630f3d-ba54-42c2-8fa4-315214cf095a\") " pod="openshift-console/console-8d454b4cf-rwfcx" Jul 16 21:48:47.200100 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:47.200080 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c3630f3d-ba54-42c2-8fa4-315214cf095a-console-config\") pod \"console-8d454b4cf-rwfcx\" (UID: \"c3630f3d-ba54-42c2-8fa4-315214cf095a\") " pod="openshift-console/console-8d454b4cf-rwfcx" Jul 16 21:48:47.200619 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:47.200599 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3630f3d-ba54-42c2-8fa4-315214cf095a-trusted-ca-bundle\") pod \"console-8d454b4cf-rwfcx\" (UID: \"c3630f3d-ba54-42c2-8fa4-315214cf095a\") " pod="openshift-console/console-8d454b4cf-rwfcx" Jul 16 21:48:47.202060 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:47.202039 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c3630f3d-ba54-42c2-8fa4-315214cf095a-console-serving-cert\") pod \"console-8d454b4cf-rwfcx\" (UID: \"c3630f3d-ba54-42c2-8fa4-315214cf095a\") " pod="openshift-console/console-8d454b4cf-rwfcx" Jul 16 21:48:47.202256 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:47.202232 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c3630f3d-ba54-42c2-8fa4-315214cf095a-console-oauth-config\") pod \"console-8d454b4cf-rwfcx\" (UID: \"c3630f3d-ba54-42c2-8fa4-315214cf095a\") " pod="openshift-console/console-8d454b4cf-rwfcx" Jul 16 21:48:47.209322 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:47.209301 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cxtn\" (UniqueName: \"kubernetes.io/projected/c3630f3d-ba54-42c2-8fa4-315214cf095a-kube-api-access-9cxtn\") pod \"console-8d454b4cf-rwfcx\" (UID: \"c3630f3d-ba54-42c2-8fa4-315214cf095a\") " pod="openshift-console/console-8d454b4cf-rwfcx" Jul 16 21:48:47.245123 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:47.245098 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8d454b4cf-rwfcx" Jul 16 21:48:48.203915 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:48.203875 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:48.339460 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:48.339420 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:48.695252 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:48.695192 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:48.883429 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:48.883306 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:48.960080 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:48.958673 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-gt52v"] Jul 16 21:48:48.968965 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:48.968925 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6ww9h"] Jul 16 21:48:48.969442 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:48.969082 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-gt52v" Jul 16 21:48:48.972808 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:48.972785 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6ww9h" Jul 16 21:48:48.972944 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:48.972840 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 16 21:48:48.973086 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:48.973064 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 16 21:48:48.973150 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:48.973115 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 16 21:48:48.973198 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:48.973066 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 16 21:48:48.973398 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:48.973378 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-fgrfz\"" Jul 16 21:48:48.973539 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:48.973521 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 16 21:48:48.973597 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:48.973524 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 16 21:48:48.977831 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:48.977299 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 16 21:48:48.977831 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:48.977598 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 16 21:48:48.987000 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:48.986495 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 16 21:48:48.987000 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:48.986829 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-w8w4k\"" Jul 16 21:48:48.988150 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:48.988123 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-gt52v"] Jul 16 21:48:49.010311 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.010278 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4ab63d4c-dc11-4ed8-8f34-4dfcce97f939-node-exporter-wtmp\") pod \"node-exporter-6ww9h\" (UID: \"4ab63d4c-dc11-4ed8-8f34-4dfcce97f939\") " pod="openshift-monitoring/node-exporter-6ww9h" Jul 16 21:48:49.010311 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.010322 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4ab63d4c-dc11-4ed8-8f34-4dfcce97f939-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6ww9h\" (UID: \"4ab63d4c-dc11-4ed8-8f34-4dfcce97f939\") " pod="openshift-monitoring/node-exporter-6ww9h" Jul 16 21:48:49.010559 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.010386 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4ab63d4c-dc11-4ed8-8f34-4dfcce97f939-node-exporter-tls\") pod \"node-exporter-6ww9h\" (UID: \"4ab63d4c-dc11-4ed8-8f34-4dfcce97f939\") " pod="openshift-monitoring/node-exporter-6ww9h" Jul 16 21:48:49.010559 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.010451 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4ab63d4c-dc11-4ed8-8f34-4dfcce97f939-node-exporter-accelerators-collector-config\") pod \"node-exporter-6ww9h\" (UID: \"4ab63d4c-dc11-4ed8-8f34-4dfcce97f939\") " pod="openshift-monitoring/node-exporter-6ww9h" Jul 16 21:48:49.010559 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.010487 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c82e3280-90dc-4a6f-858f-66024893e398-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-gt52v\" (UID: \"c82e3280-90dc-4a6f-858f-66024893e398\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-gt52v" Jul 16 21:48:49.010731 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.010568 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4ab63d4c-dc11-4ed8-8f34-4dfcce97f939-sys\") pod \"node-exporter-6ww9h\" (UID: \"4ab63d4c-dc11-4ed8-8f34-4dfcce97f939\") " pod="openshift-monitoring/node-exporter-6ww9h" Jul 16 21:48:49.010731 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.010607 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4ab63d4c-dc11-4ed8-8f34-4dfcce97f939-root\") pod \"node-exporter-6ww9h\" (UID: \"4ab63d4c-dc11-4ed8-8f34-4dfcce97f939\") " pod="openshift-monitoring/node-exporter-6ww9h" Jul 16 21:48:49.010731 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.010637 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c82e3280-90dc-4a6f-858f-66024893e398-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-gt52v\" (UID: \"c82e3280-90dc-4a6f-858f-66024893e398\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-gt52v" Jul 16 21:48:49.010731 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.010669 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c82e3280-90dc-4a6f-858f-66024893e398-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-gt52v\" (UID: \"c82e3280-90dc-4a6f-858f-66024893e398\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-gt52v" Jul 16 21:48:49.010731 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.010698 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ab63d4c-dc11-4ed8-8f34-4dfcce97f939-metrics-client-ca\") pod \"node-exporter-6ww9h\" (UID: \"4ab63d4c-dc11-4ed8-8f34-4dfcce97f939\") " pod="openshift-monitoring/node-exporter-6ww9h" Jul 16 21:48:49.010990 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.010792 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c82e3280-90dc-4a6f-858f-66024893e398-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-gt52v\" (UID: \"c82e3280-90dc-4a6f-858f-66024893e398\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-gt52v" Jul 16 21:48:49.010990 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.010826 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4ab63d4c-dc11-4ed8-8f34-4dfcce97f939-node-exporter-textfile\") pod \"node-exporter-6ww9h\" (UID: \"4ab63d4c-dc11-4ed8-8f34-4dfcce97f939\") " pod="openshift-monitoring/node-exporter-6ww9h" Jul 16 21:48:49.010990 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.010853 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbgcv\" (UniqueName: \"kubernetes.io/projected/c82e3280-90dc-4a6f-858f-66024893e398-kube-api-access-bbgcv\") pod \"kube-state-metrics-56c5594674-gt52v\" (UID: \"c82e3280-90dc-4a6f-858f-66024893e398\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-gt52v" Jul 16 21:48:49.010990 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.010921 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c82e3280-90dc-4a6f-858f-66024893e398-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-gt52v\" (UID: \"c82e3280-90dc-4a6f-858f-66024893e398\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-gt52v" Jul 16 21:48:49.010990 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.010955 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frd42\" (UniqueName: \"kubernetes.io/projected/4ab63d4c-dc11-4ed8-8f34-4dfcce97f939-kube-api-access-frd42\") pod \"node-exporter-6ww9h\" (UID: \"4ab63d4c-dc11-4ed8-8f34-4dfcce97f939\") " pod="openshift-monitoring/node-exporter-6ww9h" Jul 16 21:48:49.062738 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.062702 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:49.111315 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.111270 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c82e3280-90dc-4a6f-858f-66024893e398-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-gt52v\" (UID: \"c82e3280-90dc-4a6f-858f-66024893e398\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-gt52v" Jul 16 21:48:49.111315 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.111322 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ab63d4c-dc11-4ed8-8f34-4dfcce97f939-metrics-client-ca\") pod \"node-exporter-6ww9h\" (UID: \"4ab63d4c-dc11-4ed8-8f34-4dfcce97f939\") " pod="openshift-monitoring/node-exporter-6ww9h" Jul 16 21:48:49.111550 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.111354 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c82e3280-90dc-4a6f-858f-66024893e398-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-gt52v\" (UID: \"c82e3280-90dc-4a6f-858f-66024893e398\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-gt52v" Jul 16 21:48:49.111550 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.111384 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4ab63d4c-dc11-4ed8-8f34-4dfcce97f939-node-exporter-textfile\") pod \"node-exporter-6ww9h\" (UID: \"4ab63d4c-dc11-4ed8-8f34-4dfcce97f939\") " pod="openshift-monitoring/node-exporter-6ww9h" Jul 16 21:48:49.111550 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.111410 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbgcv\" (UniqueName: \"kubernetes.io/projected/c82e3280-90dc-4a6f-858f-66024893e398-kube-api-access-bbgcv\") pod \"kube-state-metrics-56c5594674-gt52v\" (UID: \"c82e3280-90dc-4a6f-858f-66024893e398\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-gt52v" Jul 16 21:48:49.111550 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:49.111467 2585 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 16 21:48:49.111550 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:48:49.111541 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c82e3280-90dc-4a6f-858f-66024893e398-kube-state-metrics-tls podName:c82e3280-90dc-4a6f-858f-66024893e398 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:49.611519998 +0000 UTC m=+78.233962928 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/c82e3280-90dc-4a6f-858f-66024893e398-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-gt52v" (UID: "c82e3280-90dc-4a6f-858f-66024893e398") : secret "kube-state-metrics-tls" not found Jul 16 21:48:49.111715 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.111678 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c82e3280-90dc-4a6f-858f-66024893e398-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-gt52v\" (UID: \"c82e3280-90dc-4a6f-858f-66024893e398\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-gt52v" Jul 16 21:48:49.111770 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.111719 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frd42\" (UniqueName: \"kubernetes.io/projected/4ab63d4c-dc11-4ed8-8f34-4dfcce97f939-kube-api-access-frd42\") pod \"node-exporter-6ww9h\" (UID: \"4ab63d4c-dc11-4ed8-8f34-4dfcce97f939\") " pod="openshift-monitoring/node-exporter-6ww9h" Jul 16 21:48:49.111770 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.111750 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4ab63d4c-dc11-4ed8-8f34-4dfcce97f939-node-exporter-wtmp\") pod \"node-exporter-6ww9h\" (UID: \"4ab63d4c-dc11-4ed8-8f34-4dfcce97f939\") " pod="openshift-monitoring/node-exporter-6ww9h" Jul 16 21:48:49.111868 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.111776 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4ab63d4c-dc11-4ed8-8f34-4dfcce97f939-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6ww9h\" (UID: \"4ab63d4c-dc11-4ed8-8f34-4dfcce97f939\") " pod="openshift-monitoring/node-exporter-6ww9h" Jul 16 21:48:49.111868 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.111827 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4ab63d4c-dc11-4ed8-8f34-4dfcce97f939-node-exporter-tls\") pod \"node-exporter-6ww9h\" (UID: \"4ab63d4c-dc11-4ed8-8f34-4dfcce97f939\") " pod="openshift-monitoring/node-exporter-6ww9h" Jul 16 21:48:49.111868 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.111857 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4ab63d4c-dc11-4ed8-8f34-4dfcce97f939-node-exporter-accelerators-collector-config\") pod \"node-exporter-6ww9h\" (UID: \"4ab63d4c-dc11-4ed8-8f34-4dfcce97f939\") " pod="openshift-monitoring/node-exporter-6ww9h" Jul 16 21:48:49.112019 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.111885 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c82e3280-90dc-4a6f-858f-66024893e398-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-gt52v\" (UID: \"c82e3280-90dc-4a6f-858f-66024893e398\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-gt52v" Jul 16 21:48:49.112019 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.111925 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4ab63d4c-dc11-4ed8-8f34-4dfcce97f939-sys\") pod \"node-exporter-6ww9h\" (UID: \"4ab63d4c-dc11-4ed8-8f34-4dfcce97f939\") " pod="openshift-monitoring/node-exporter-6ww9h" Jul 16 21:48:49.112019 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.111961 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4ab63d4c-dc11-4ed8-8f34-4dfcce97f939-root\") pod \"node-exporter-6ww9h\" (UID: \"4ab63d4c-dc11-4ed8-8f34-4dfcce97f939\") " pod="openshift-monitoring/node-exporter-6ww9h" Jul 16 21:48:49.112019 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.111989 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ab63d4c-dc11-4ed8-8f34-4dfcce97f939-metrics-client-ca\") pod \"node-exporter-6ww9h\" (UID: \"4ab63d4c-dc11-4ed8-8f34-4dfcce97f939\") " pod="openshift-monitoring/node-exporter-6ww9h" Jul 16 21:48:49.112204 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.111993 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c82e3280-90dc-4a6f-858f-66024893e398-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-gt52v\" (UID: \"c82e3280-90dc-4a6f-858f-66024893e398\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-gt52v" Jul 16 21:48:49.112204 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.112048 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4ab63d4c-dc11-4ed8-8f34-4dfcce97f939-node-exporter-textfile\") pod \"node-exporter-6ww9h\" (UID: \"4ab63d4c-dc11-4ed8-8f34-4dfcce97f939\") " pod="openshift-monitoring/node-exporter-6ww9h" Jul 16 21:48:49.112204 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.112160 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4ab63d4c-dc11-4ed8-8f34-4dfcce97f939-sys\") pod \"node-exporter-6ww9h\" (UID: \"4ab63d4c-dc11-4ed8-8f34-4dfcce97f939\") " pod="openshift-monitoring/node-exporter-6ww9h" Jul 16 21:48:49.112387 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.112289 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4ab63d4c-dc11-4ed8-8f34-4dfcce97f939-root\") pod \"node-exporter-6ww9h\" (UID: \"4ab63d4c-dc11-4ed8-8f34-4dfcce97f939\") " pod="openshift-monitoring/node-exporter-6ww9h" Jul 16 21:48:49.112477 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.112453 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c82e3280-90dc-4a6f-858f-66024893e398-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-gt52v\" (UID: \"c82e3280-90dc-4a6f-858f-66024893e398\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-gt52v" Jul 16 21:48:49.112535 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.112504 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c82e3280-90dc-4a6f-858f-66024893e398-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-gt52v\" (UID: \"c82e3280-90dc-4a6f-858f-66024893e398\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-gt52v" Jul 16 21:48:49.112614 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.112593 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c82e3280-90dc-4a6f-858f-66024893e398-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-gt52v\" (UID: \"c82e3280-90dc-4a6f-858f-66024893e398\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-gt52v" Jul 16 21:48:49.112692 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.112594 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4ab63d4c-dc11-4ed8-8f34-4dfcce97f939-node-exporter-accelerators-collector-config\") pod \"node-exporter-6ww9h\" (UID: \"4ab63d4c-dc11-4ed8-8f34-4dfcce97f939\") " pod="openshift-monitoring/node-exporter-6ww9h" Jul 16 21:48:49.112692 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.112647 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4ab63d4c-dc11-4ed8-8f34-4dfcce97f939-node-exporter-wtmp\") pod \"node-exporter-6ww9h\" (UID: \"4ab63d4c-dc11-4ed8-8f34-4dfcce97f939\") " pod="openshift-monitoring/node-exporter-6ww9h" Jul 16 21:48:49.114462 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.114419 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c82e3280-90dc-4a6f-858f-66024893e398-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-gt52v\" (UID: \"c82e3280-90dc-4a6f-858f-66024893e398\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-gt52v" Jul 16 21:48:49.114864 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.114822 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4ab63d4c-dc11-4ed8-8f34-4dfcce97f939-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6ww9h\" (UID: \"4ab63d4c-dc11-4ed8-8f34-4dfcce97f939\") " pod="openshift-monitoring/node-exporter-6ww9h" Jul 16 21:48:49.115043 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.115024 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4ab63d4c-dc11-4ed8-8f34-4dfcce97f939-node-exporter-tls\") pod \"node-exporter-6ww9h\" (UID: \"4ab63d4c-dc11-4ed8-8f34-4dfcce97f939\") " pod="openshift-monitoring/node-exporter-6ww9h" Jul 16 21:48:49.124206 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.124182 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frd42\" (UniqueName: \"kubernetes.io/projected/4ab63d4c-dc11-4ed8-8f34-4dfcce97f939-kube-api-access-frd42\") pod \"node-exporter-6ww9h\" (UID: \"4ab63d4c-dc11-4ed8-8f34-4dfcce97f939\") " pod="openshift-monitoring/node-exporter-6ww9h" Jul 16 21:48:49.124375 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.124349 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbgcv\" (UniqueName: \"kubernetes.io/projected/c82e3280-90dc-4a6f-858f-66024893e398-kube-api-access-bbgcv\") pod \"kube-state-metrics-56c5594674-gt52v\" (UID: \"c82e3280-90dc-4a6f-858f-66024893e398\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-gt52v" Jul 16 21:48:49.163121 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.163083 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:49.292440 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.292367 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6ww9h" Jul 16 21:48:49.353195 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.353165 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:49.514325 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.514282 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:49.613428 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.613394 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:49.614450 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.614421 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c82e3280-90dc-4a6f-858f-66024893e398-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-gt52v\" (UID: \"c82e3280-90dc-4a6f-858f-66024893e398\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-gt52v" Jul 16 21:48:49.617091 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.617068 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c82e3280-90dc-4a6f-858f-66024893e398-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-gt52v\" (UID: \"c82e3280-90dc-4a6f-858f-66024893e398\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-gt52v" Jul 16 21:48:49.887047 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:49.886969 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-gt52v" Jul 16 21:48:50.028548 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.028501 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:50.146434 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.146339 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:50.213701 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.213667 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 21:48:50.264698 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.264650 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:50.275249 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.275207 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 21:48:50.275405 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.275367 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:48:50.281828 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.281791 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 16 21:48:50.281997 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.281803 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 16 21:48:50.282230 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.282203 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-5f4vw\"" Jul 16 21:48:50.282656 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.282625 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 16 21:48:50.283094 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.283024 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 16 21:48:50.284263 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.284245 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 16 21:48:50.284473 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.284455 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 16 21:48:50.284473 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.284462 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 16 21:48:50.284634 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.284478 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 16 21:48:50.296809 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.296778 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 16 21:48:50.320891 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.320858 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6a0419d8-50b3-4681-84a5-500b6c56fcb9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:48:50.321026 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.320911 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6a0419d8-50b3-4681-84a5-500b6c56fcb9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:48:50.321026 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.320945 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6a0419d8-50b3-4681-84a5-500b6c56fcb9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:48:50.321026 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.320974 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a0419d8-50b3-4681-84a5-500b6c56fcb9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:48:50.321026 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.320992 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6a0419d8-50b3-4681-84a5-500b6c56fcb9-config-out\") pod \"alertmanager-main-0\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:48:50.321207 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.321047 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6a0419d8-50b3-4681-84a5-500b6c56fcb9-web-config\") pod \"alertmanager-main-0\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:48:50.321207 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.321100 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a0419d8-50b3-4681-84a5-500b6c56fcb9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:48:50.321207 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.321126 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z9zw\" (UniqueName: \"kubernetes.io/projected/6a0419d8-50b3-4681-84a5-500b6c56fcb9-kube-api-access-6z9zw\") pod \"alertmanager-main-0\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:48:50.321207 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.321164 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6a0419d8-50b3-4681-84a5-500b6c56fcb9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:48:50.321422 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.321246 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6a0419d8-50b3-4681-84a5-500b6c56fcb9-config-volume\") pod \"alertmanager-main-0\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:48:50.321422 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.321288 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6a0419d8-50b3-4681-84a5-500b6c56fcb9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:48:50.321422 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.321316 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6a0419d8-50b3-4681-84a5-500b6c56fcb9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:48:50.321422 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.321340 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6a0419d8-50b3-4681-84a5-500b6c56fcb9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:48:50.422092 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.422011 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6a0419d8-50b3-4681-84a5-500b6c56fcb9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:48:50.422092 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.422059 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6a0419d8-50b3-4681-84a5-500b6c56fcb9-config-volume\") pod \"alertmanager-main-0\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:48:50.422092 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.422092 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6a0419d8-50b3-4681-84a5-500b6c56fcb9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:48:50.422403 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.422123 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6a0419d8-50b3-4681-84a5-500b6c56fcb9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:48:50.422403 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.422152 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6a0419d8-50b3-4681-84a5-500b6c56fcb9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:48:50.422403 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.422206 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6a0419d8-50b3-4681-84a5-500b6c56fcb9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:48:50.422403 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.422253 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6a0419d8-50b3-4681-84a5-500b6c56fcb9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:48:50.422403 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.422282 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6a0419d8-50b3-4681-84a5-500b6c56fcb9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:48:50.422403 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.422322 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a0419d8-50b3-4681-84a5-500b6c56fcb9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:48:50.422403 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.422349 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6a0419d8-50b3-4681-84a5-500b6c56fcb9-config-out\") pod \"alertmanager-main-0\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:48:50.422403 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.422378 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6a0419d8-50b3-4681-84a5-500b6c56fcb9-web-config\") pod \"alertmanager-main-0\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:48:50.422769 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.422413 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a0419d8-50b3-4681-84a5-500b6c56fcb9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:48:50.422769 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.422437 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z9zw\" (UniqueName: \"kubernetes.io/projected/6a0419d8-50b3-4681-84a5-500b6c56fcb9-kube-api-access-6z9zw\") pod \"alertmanager-main-0\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:48:50.423327 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.423243 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6a0419d8-50b3-4681-84a5-500b6c56fcb9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:48:50.424204 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.423912 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a0419d8-50b3-4681-84a5-500b6c56fcb9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:48:50.424934 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.424906 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a0419d8-50b3-4681-84a5-500b6c56fcb9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:48:50.425323 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.425279 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6a0419d8-50b3-4681-84a5-500b6c56fcb9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:48:50.425323 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.425313 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6a0419d8-50b3-4681-84a5-500b6c56fcb9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:48:50.426552 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.426526 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6a0419d8-50b3-4681-84a5-500b6c56fcb9-config-out\") pod \"alertmanager-main-0\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:48:50.426662 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.426530 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6a0419d8-50b3-4681-84a5-500b6c56fcb9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:48:50.426662 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.426629 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6a0419d8-50b3-4681-84a5-500b6c56fcb9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:48:50.427311 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.427195 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6a0419d8-50b3-4681-84a5-500b6c56fcb9-config-volume\") pod \"alertmanager-main-0\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:48:50.427541 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.427513 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6a0419d8-50b3-4681-84a5-500b6c56fcb9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:48:50.427837 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.427819 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6a0419d8-50b3-4681-84a5-500b6c56fcb9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:48:50.428036 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.428013 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6a0419d8-50b3-4681-84a5-500b6c56fcb9-web-config\") pod \"alertmanager-main-0\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:48:50.437200 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.437175 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z9zw\" (UniqueName: \"kubernetes.io/projected/6a0419d8-50b3-4681-84a5-500b6c56fcb9-kube-api-access-6z9zw\") pod \"alertmanager-main-0\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:48:50.482810 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.482778 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:50.587004 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:50.586968 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:48:51.578714 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:51.578673 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:51.776158 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:51.776114 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:51.876014 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:51.875934 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:52.166237 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:52.166113 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:52.264396 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:52.264333 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:52.367399 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:52.367353 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:52.667332 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:52.667290 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:53.112018 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:48:53.111952 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ab63d4c_dc11_4ed8_8f34_4dfcce97f939.slice/crio-edb237986ddd903825e79f3cc7c5ed34ccf2f4c8a762ab78b9c09c1ac7637c26 WatchSource:0}: Error finding container edb237986ddd903825e79f3cc7c5ed34ccf2f4c8a762ab78b9c09c1ac7637c26: Status 404 returned error can't find the container with id edb237986ddd903825e79f3cc7c5ed34ccf2f4c8a762ab78b9c09c1ac7637c26 Jul 16 21:48:53.275479 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.275452 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 21:48:53.282162 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.282107 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-gt52v"] Jul 16 21:48:53.283078 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.283054 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6ww9h" event={"ID":"4ab63d4c-dc11-4ed8-8f34-4dfcce97f939","Type":"ContainerStarted","Data":"edb237986ddd903825e79f3cc7c5ed34ccf2f4c8a762ab78b9c09c1ac7637c26"} Jul 16 21:48:53.293316 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.293251 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8d454b4cf-rwfcx"] Jul 16 21:48:53.399212 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.399142 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5d897c97d7-c8rk6"] Jul 16 21:48:53.413550 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.413521 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5d897c97d7-c8rk6" Jul 16 21:48:53.416203 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.416169 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 16 21:48:53.416353 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.416289 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-4lhj03844n49o\"" Jul 16 21:48:53.416353 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.416298 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 16 21:48:53.416952 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.416938 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-lxdtf\"" Jul 16 21:48:53.417025 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.416962 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 16 21:48:53.417025 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.417006 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 16 21:48:53.421589 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:48:53.421563 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0419d8_50b3_4681_84a5_500b6c56fcb9.slice/crio-b56c1f01e5a3b1aea5bb2e04ed8b42c05c1149bc5e079bc20eeefadf218f4bbf WatchSource:0}: Error finding container b56c1f01e5a3b1aea5bb2e04ed8b42c05c1149bc5e079bc20eeefadf218f4bbf: Status 404 returned error can't find the container with id b56c1f01e5a3b1aea5bb2e04ed8b42c05c1149bc5e079bc20eeefadf218f4bbf Jul 16 21:48:53.423180 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:48:53.423122 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc82e3280_90dc_4a6f_858f_66024893e398.slice/crio-f869a90b887a023ff41899bc1e28582a90bbd7e06605e8ebca782a424087e298 WatchSource:0}: Error finding container f869a90b887a023ff41899bc1e28582a90bbd7e06605e8ebca782a424087e298: Status 404 returned error can't find the container with id f869a90b887a023ff41899bc1e28582a90bbd7e06605e8ebca782a424087e298 Jul 16 21:48:53.424301 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:48:53.424274 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3630f3d_ba54_42c2_8fa4_315214cf095a.slice/crio-6f085e88365bdf05137c7891f1a25a6c466c02422acc6cc9fa9d9e4ade393194 WatchSource:0}: Error finding container 6f085e88365bdf05137c7891f1a25a6c466c02422acc6cc9fa9d9e4ade393194: Status 404 returned error can't find the container with id 6f085e88365bdf05137c7891f1a25a6c466c02422acc6cc9fa9d9e4ade393194 Jul 16 21:48:53.424805 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.424422 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5d897c97d7-c8rk6"] Jul 16 21:48:53.447287 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.447260 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5485377e-20a2-4f1e-b335-8eed2032a4bf-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5d897c97d7-c8rk6\" (UID: \"5485377e-20a2-4f1e-b335-8eed2032a4bf\") " pod="openshift-monitoring/metrics-server-5d897c97d7-c8rk6" Jul 16 21:48:53.447893 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.447322 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5485377e-20a2-4f1e-b335-8eed2032a4bf-audit-log\") pod \"metrics-server-5d897c97d7-c8rk6\" (UID: \"5485377e-20a2-4f1e-b335-8eed2032a4bf\") " pod="openshift-monitoring/metrics-server-5d897c97d7-c8rk6" Jul 16 21:48:53.447893 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.447812 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bg2w\" (UniqueName: \"kubernetes.io/projected/5485377e-20a2-4f1e-b335-8eed2032a4bf-kube-api-access-8bg2w\") pod \"metrics-server-5d897c97d7-c8rk6\" (UID: \"5485377e-20a2-4f1e-b335-8eed2032a4bf\") " pod="openshift-monitoring/metrics-server-5d897c97d7-c8rk6" Jul 16 21:48:53.447893 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.447861 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5485377e-20a2-4f1e-b335-8eed2032a4bf-client-ca-bundle\") pod \"metrics-server-5d897c97d7-c8rk6\" (UID: \"5485377e-20a2-4f1e-b335-8eed2032a4bf\") " pod="openshift-monitoring/metrics-server-5d897c97d7-c8rk6" Jul 16 21:48:53.448152 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.447974 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5485377e-20a2-4f1e-b335-8eed2032a4bf-secret-metrics-server-tls\") pod \"metrics-server-5d897c97d7-c8rk6\" (UID: \"5485377e-20a2-4f1e-b335-8eed2032a4bf\") " pod="openshift-monitoring/metrics-server-5d897c97d7-c8rk6" Jul 16 21:48:53.448152 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.448088 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5485377e-20a2-4f1e-b335-8eed2032a4bf-metrics-server-audit-profiles\") pod \"metrics-server-5d897c97d7-c8rk6\" (UID: \"5485377e-20a2-4f1e-b335-8eed2032a4bf\") " pod="openshift-monitoring/metrics-server-5d897c97d7-c8rk6" Jul 16 21:48:53.448152 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.448132 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5485377e-20a2-4f1e-b335-8eed2032a4bf-secret-metrics-server-client-certs\") pod \"metrics-server-5d897c97d7-c8rk6\" (UID: \"5485377e-20a2-4f1e-b335-8eed2032a4bf\") " pod="openshift-monitoring/metrics-server-5d897c97d7-c8rk6" Jul 16 21:48:53.549145 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.549108 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5485377e-20a2-4f1e-b335-8eed2032a4bf-audit-log\") pod \"metrics-server-5d897c97d7-c8rk6\" (UID: \"5485377e-20a2-4f1e-b335-8eed2032a4bf\") " pod="openshift-monitoring/metrics-server-5d897c97d7-c8rk6" Jul 16 21:48:53.549145 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.549148 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bg2w\" (UniqueName: \"kubernetes.io/projected/5485377e-20a2-4f1e-b335-8eed2032a4bf-kube-api-access-8bg2w\") pod \"metrics-server-5d897c97d7-c8rk6\" (UID: \"5485377e-20a2-4f1e-b335-8eed2032a4bf\") " pod="openshift-monitoring/metrics-server-5d897c97d7-c8rk6" Jul 16 21:48:53.549390 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.549167 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5485377e-20a2-4f1e-b335-8eed2032a4bf-client-ca-bundle\") pod \"metrics-server-5d897c97d7-c8rk6\" (UID: \"5485377e-20a2-4f1e-b335-8eed2032a4bf\") " pod="openshift-monitoring/metrics-server-5d897c97d7-c8rk6" Jul 16 21:48:53.549390 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.549190 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5485377e-20a2-4f1e-b335-8eed2032a4bf-secret-metrics-server-tls\") pod \"metrics-server-5d897c97d7-c8rk6\" (UID: \"5485377e-20a2-4f1e-b335-8eed2032a4bf\") " pod="openshift-monitoring/metrics-server-5d897c97d7-c8rk6" Jul 16 21:48:53.549390 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.549265 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5485377e-20a2-4f1e-b335-8eed2032a4bf-metrics-server-audit-profiles\") pod \"metrics-server-5d897c97d7-c8rk6\" (UID: \"5485377e-20a2-4f1e-b335-8eed2032a4bf\") " pod="openshift-monitoring/metrics-server-5d897c97d7-c8rk6" Jul 16 21:48:53.549390 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.549286 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5485377e-20a2-4f1e-b335-8eed2032a4bf-secret-metrics-server-client-certs\") pod \"metrics-server-5d897c97d7-c8rk6\" (UID: \"5485377e-20a2-4f1e-b335-8eed2032a4bf\") " pod="openshift-monitoring/metrics-server-5d897c97d7-c8rk6" Jul 16 21:48:53.549390 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.549311 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5485377e-20a2-4f1e-b335-8eed2032a4bf-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5d897c97d7-c8rk6\" (UID: \"5485377e-20a2-4f1e-b335-8eed2032a4bf\") " pod="openshift-monitoring/metrics-server-5d897c97d7-c8rk6" Jul 16 21:48:53.549603 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.549585 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5485377e-20a2-4f1e-b335-8eed2032a4bf-audit-log\") pod \"metrics-server-5d897c97d7-c8rk6\" (UID: \"5485377e-20a2-4f1e-b335-8eed2032a4bf\") " pod="openshift-monitoring/metrics-server-5d897c97d7-c8rk6" Jul 16 21:48:53.550052 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.550024 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5485377e-20a2-4f1e-b335-8eed2032a4bf-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5d897c97d7-c8rk6\" (UID: \"5485377e-20a2-4f1e-b335-8eed2032a4bf\") " pod="openshift-monitoring/metrics-server-5d897c97d7-c8rk6" Jul 16 21:48:53.550288 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.550269 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5485377e-20a2-4f1e-b335-8eed2032a4bf-metrics-server-audit-profiles\") pod \"metrics-server-5d897c97d7-c8rk6\" (UID: \"5485377e-20a2-4f1e-b335-8eed2032a4bf\") " pod="openshift-monitoring/metrics-server-5d897c97d7-c8rk6" Jul 16 21:48:53.551721 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.551699 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5485377e-20a2-4f1e-b335-8eed2032a4bf-secret-metrics-server-client-certs\") pod \"metrics-server-5d897c97d7-c8rk6\" (UID: \"5485377e-20a2-4f1e-b335-8eed2032a4bf\") " pod="openshift-monitoring/metrics-server-5d897c97d7-c8rk6" Jul 16 21:48:53.551802 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.551791 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5485377e-20a2-4f1e-b335-8eed2032a4bf-client-ca-bundle\") pod \"metrics-server-5d897c97d7-c8rk6\" (UID: \"5485377e-20a2-4f1e-b335-8eed2032a4bf\") " pod="openshift-monitoring/metrics-server-5d897c97d7-c8rk6" Jul 16 21:48:53.551843 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.551814 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5485377e-20a2-4f1e-b335-8eed2032a4bf-secret-metrics-server-tls\") pod \"metrics-server-5d897c97d7-c8rk6\" (UID: \"5485377e-20a2-4f1e-b335-8eed2032a4bf\") " pod="openshift-monitoring/metrics-server-5d897c97d7-c8rk6" Jul 16 21:48:53.559326 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.559301 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bg2w\" (UniqueName: \"kubernetes.io/projected/5485377e-20a2-4f1e-b335-8eed2032a4bf-kube-api-access-8bg2w\") pod \"metrics-server-5d897c97d7-c8rk6\" (UID: \"5485377e-20a2-4f1e-b335-8eed2032a4bf\") " pod="openshift-monitoring/metrics-server-5d897c97d7-c8rk6" Jul 16 21:48:53.716862 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.716765 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:53.738883 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.738857 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5d897c97d7-c8rk6" Jul 16 21:48:53.780463 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.780439 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8d454b4cf-rwfcx"] Jul 16 21:48:53.825324 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.825013 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-bbc77df58-9mpjq"] Jul 16 21:48:53.826671 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.826333 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:53.827341 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.827321 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bbc77df58-9mpjq" Jul 16 21:48:53.852493 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.852467 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bbc77df58-9mpjq"] Jul 16 21:48:53.856706 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.856670 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/74324078-6d2f-4256-bd15-150bf88c8f0d-console-serving-cert\") pod \"console-bbc77df58-9mpjq\" (UID: \"74324078-6d2f-4256-bd15-150bf88c8f0d\") " pod="openshift-console/console-bbc77df58-9mpjq" Jul 16 21:48:53.856823 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.856729 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/74324078-6d2f-4256-bd15-150bf88c8f0d-console-oauth-config\") pod \"console-bbc77df58-9mpjq\" (UID: \"74324078-6d2f-4256-bd15-150bf88c8f0d\") " pod="openshift-console/console-bbc77df58-9mpjq" Jul 16 21:48:53.856823 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.856769 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/74324078-6d2f-4256-bd15-150bf88c8f0d-console-config\") pod \"console-bbc77df58-9mpjq\" (UID: \"74324078-6d2f-4256-bd15-150bf88c8f0d\") " pod="openshift-console/console-bbc77df58-9mpjq" Jul 16 21:48:53.856823 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.856804 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74324078-6d2f-4256-bd15-150bf88c8f0d-trusted-ca-bundle\") pod \"console-bbc77df58-9mpjq\" (UID: \"74324078-6d2f-4256-bd15-150bf88c8f0d\") " pod="openshift-console/console-bbc77df58-9mpjq" Jul 16 21:48:53.857035 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.856834 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/74324078-6d2f-4256-bd15-150bf88c8f0d-oauth-serving-cert\") pod \"console-bbc77df58-9mpjq\" (UID: \"74324078-6d2f-4256-bd15-150bf88c8f0d\") " pod="openshift-console/console-bbc77df58-9mpjq" Jul 16 21:48:53.857035 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.856877 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsfr2\" (UniqueName: \"kubernetes.io/projected/74324078-6d2f-4256-bd15-150bf88c8f0d-kube-api-access-tsfr2\") pod \"console-bbc77df58-9mpjq\" (UID: \"74324078-6d2f-4256-bd15-150bf88c8f0d\") " pod="openshift-console/console-bbc77df58-9mpjq" Jul 16 21:48:53.857035 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.856951 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/74324078-6d2f-4256-bd15-150bf88c8f0d-service-ca\") pod \"console-bbc77df58-9mpjq\" (UID: \"74324078-6d2f-4256-bd15-150bf88c8f0d\") " pod="openshift-console/console-bbc77df58-9mpjq" Jul 16 21:48:53.918123 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.918088 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5d897c97d7-c8rk6"] Jul 16 21:48:53.925042 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:48:53.924982 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5485377e_20a2_4f1e_b335_8eed2032a4bf.slice/crio-05ad97e63fcee4525359aac242568173d6139865e566094e77e3699e02c6e580 WatchSource:0}: Error finding container 05ad97e63fcee4525359aac242568173d6139865e566094e77e3699e02c6e580: Status 404 returned error can't find the container with id 05ad97e63fcee4525359aac242568173d6139865e566094e77e3699e02c6e580 Jul 16 21:48:53.927517 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.927457 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:53.959470 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.958957 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74324078-6d2f-4256-bd15-150bf88c8f0d-trusted-ca-bundle\") pod \"console-bbc77df58-9mpjq\" (UID: \"74324078-6d2f-4256-bd15-150bf88c8f0d\") " pod="openshift-console/console-bbc77df58-9mpjq" Jul 16 21:48:53.959470 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.959012 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/74324078-6d2f-4256-bd15-150bf88c8f0d-oauth-serving-cert\") pod \"console-bbc77df58-9mpjq\" (UID: \"74324078-6d2f-4256-bd15-150bf88c8f0d\") " pod="openshift-console/console-bbc77df58-9mpjq" Jul 16 21:48:53.959470 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.959065 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsfr2\" (UniqueName: \"kubernetes.io/projected/74324078-6d2f-4256-bd15-150bf88c8f0d-kube-api-access-tsfr2\") pod \"console-bbc77df58-9mpjq\" (UID: \"74324078-6d2f-4256-bd15-150bf88c8f0d\") " pod="openshift-console/console-bbc77df58-9mpjq" Jul 16 21:48:53.959772 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.959595 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/74324078-6d2f-4256-bd15-150bf88c8f0d-service-ca\") pod \"console-bbc77df58-9mpjq\" (UID: \"74324078-6d2f-4256-bd15-150bf88c8f0d\") " pod="openshift-console/console-bbc77df58-9mpjq" Jul 16 21:48:53.959772 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.959656 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/74324078-6d2f-4256-bd15-150bf88c8f0d-console-serving-cert\") pod \"console-bbc77df58-9mpjq\" (UID: \"74324078-6d2f-4256-bd15-150bf88c8f0d\") " pod="openshift-console/console-bbc77df58-9mpjq" Jul 16 21:48:53.959772 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.959685 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/74324078-6d2f-4256-bd15-150bf88c8f0d-console-oauth-config\") pod \"console-bbc77df58-9mpjq\" (UID: \"74324078-6d2f-4256-bd15-150bf88c8f0d\") " pod="openshift-console/console-bbc77df58-9mpjq" Jul 16 21:48:53.959772 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.959732 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/74324078-6d2f-4256-bd15-150bf88c8f0d-console-config\") pod \"console-bbc77df58-9mpjq\" (UID: \"74324078-6d2f-4256-bd15-150bf88c8f0d\") " pod="openshift-console/console-bbc77df58-9mpjq" Jul 16 21:48:53.960008 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.959940 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74324078-6d2f-4256-bd15-150bf88c8f0d-trusted-ca-bundle\") pod \"console-bbc77df58-9mpjq\" (UID: \"74324078-6d2f-4256-bd15-150bf88c8f0d\") " pod="openshift-console/console-bbc77df58-9mpjq" Jul 16 21:48:53.961697 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.961673 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/74324078-6d2f-4256-bd15-150bf88c8f0d-service-ca\") pod \"console-bbc77df58-9mpjq\" (UID: \"74324078-6d2f-4256-bd15-150bf88c8f0d\") " pod="openshift-console/console-bbc77df58-9mpjq" Jul 16 21:48:53.961993 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.961947 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/74324078-6d2f-4256-bd15-150bf88c8f0d-console-config\") pod \"console-bbc77df58-9mpjq\" (UID: \"74324078-6d2f-4256-bd15-150bf88c8f0d\") " pod="openshift-console/console-bbc77df58-9mpjq" Jul 16 21:48:53.962157 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.962116 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/74324078-6d2f-4256-bd15-150bf88c8f0d-oauth-serving-cert\") pod \"console-bbc77df58-9mpjq\" (UID: \"74324078-6d2f-4256-bd15-150bf88c8f0d\") " pod="openshift-console/console-bbc77df58-9mpjq" Jul 16 21:48:53.964202 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.964123 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/74324078-6d2f-4256-bd15-150bf88c8f0d-console-serving-cert\") pod \"console-bbc77df58-9mpjq\" (UID: \"74324078-6d2f-4256-bd15-150bf88c8f0d\") " pod="openshift-console/console-bbc77df58-9mpjq" Jul 16 21:48:53.964646 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.964604 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/74324078-6d2f-4256-bd15-150bf88c8f0d-console-oauth-config\") pod \"console-bbc77df58-9mpjq\" (UID: \"74324078-6d2f-4256-bd15-150bf88c8f0d\") " pod="openshift-console/console-bbc77df58-9mpjq" Jul 16 21:48:53.970133 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:53.970056 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsfr2\" (UniqueName: \"kubernetes.io/projected/74324078-6d2f-4256-bd15-150bf88c8f0d-kube-api-access-tsfr2\") pod \"console-bbc77df58-9mpjq\" (UID: \"74324078-6d2f-4256-bd15-150bf88c8f0d\") " pod="openshift-console/console-bbc77df58-9mpjq" Jul 16 21:48:54.139092 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:54.139053 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bbc77df58-9mpjq" Jul 16 21:48:54.295656 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:54.295613 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-njjsn" event={"ID":"68e6c5e2-0a59-422d-a1c8-9b735597ed43","Type":"ContainerStarted","Data":"fe8312d25ba47f03170a142c75b879ab3846cf065e0924506fd96350ff283641"} Jul 16 21:48:54.296725 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:54.296700 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-njjsn" Jul 16 21:48:54.299184 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:54.299073 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5d897c97d7-c8rk6" event={"ID":"5485377e-20a2-4f1e-b335-8eed2032a4bf","Type":"ContainerStarted","Data":"05ad97e63fcee4525359aac242568173d6139865e566094e77e3699e02c6e580"} Jul 16 21:48:54.300939 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:54.300853 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-gt52v" event={"ID":"c82e3280-90dc-4a6f-858f-66024893e398","Type":"ContainerStarted","Data":"f869a90b887a023ff41899bc1e28582a90bbd7e06605e8ebca782a424087e298"} Jul 16 21:48:54.302603 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:54.302095 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-njjsn" Jul 16 21:48:54.303718 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:54.303476 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a0419d8-50b3-4681-84a5-500b6c56fcb9","Type":"ContainerStarted","Data":"b56c1f01e5a3b1aea5bb2e04ed8b42c05c1149bc5e079bc20eeefadf218f4bbf"} Jul 16 21:48:54.306337 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:54.306312 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8d454b4cf-rwfcx" event={"ID":"c3630f3d-ba54-42c2-8fa4-315214cf095a","Type":"ContainerStarted","Data":"6f085e88365bdf05137c7891f1a25a6c466c02422acc6cc9fa9d9e4ade393194"} Jul 16 21:48:54.371042 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:54.370618 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-njjsn" podStartSLOduration=2.993347668 podStartE2EDuration="22.370595332s" totalImagesPullingTime="19.377247664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:54.332664031 +0000 UTC m=+82.955106985" watchObservedRunningTime="2026-07-16 21:48:54.370595332 +0000 UTC m=+82.993038285" Jul 16 21:48:54.371042 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:54.370972 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bbc77df58-9mpjq"] Jul 16 21:48:54.529836 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:54.529678 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:54.673517 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:54.673467 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:55.471089 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:48:55.471048 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74324078_6d2f_4256_bd15_150bf88c8f0d.slice/crio-9ab854505f848cadf125e67b2f6671f47b4234dc71846b09799624f8cca685d2 WatchSource:0}: Error finding container 9ab854505f848cadf125e67b2f6671f47b4234dc71846b09799624f8cca685d2: Status 404 returned error can't find the container with id 9ab854505f848cadf125e67b2f6671f47b4234dc71846b09799624f8cca685d2 Jul 16 21:48:55.627141 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:55.627092 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:56.318703 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:56.318610 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bbc77df58-9mpjq" event={"ID":"74324078-6d2f-4256-bd15-150bf88c8f0d","Type":"ContainerStarted","Data":"9ab854505f848cadf125e67b2f6671f47b4234dc71846b09799624f8cca685d2"} Jul 16 21:48:57.203351 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:57.203314 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-55599968c8-d5h5x" Jul 16 21:48:57.393350 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:57.393309 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:57.578019 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:57.577979 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:57.832386 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:57.832273 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:58.011567 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:58.011495 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:58.113161 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:58.113011 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:58.582694 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:58.582656 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:58.704775 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:58.704744 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:48:59.336357 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:59.335586 2585 generic.go:356] "Generic (PLEG): container finished" podID="6a0419d8-50b3-4681-84a5-500b6c56fcb9" containerID="470ef4afef9b1a707ee8f28b470232db59e90b78df88c405cdd8ad6e74a98046" exitCode=0 Jul 16 21:48:59.336357 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:59.335699 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a0419d8-50b3-4681-84a5-500b6c56fcb9","Type":"ContainerDied","Data":"470ef4afef9b1a707ee8f28b470232db59e90b78df88c405cdd8ad6e74a98046"} Jul 16 21:48:59.339463 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:59.338994 2585 generic.go:356] "Generic (PLEG): container finished" podID="4ab63d4c-dc11-4ed8-8f34-4dfcce97f939" containerID="110e58da499164a3dd9f048472a9ad1b31b8b3b63935c346f2ff69a0a9be0bf9" exitCode=0 Jul 16 21:48:59.339463 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:59.339111 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6ww9h" event={"ID":"4ab63d4c-dc11-4ed8-8f34-4dfcce97f939","Type":"ContainerDied","Data":"110e58da499164a3dd9f048472a9ad1b31b8b3b63935c346f2ff69a0a9be0bf9"} Jul 16 21:48:59.344974 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:59.344948 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-gt52v" event={"ID":"c82e3280-90dc-4a6f-858f-66024893e398","Type":"ContainerStarted","Data":"13268f83beea938891b5aee2fe4aefe1ba362a2966cd27ec71ce2683c8b34176"} Jul 16 21:48:59.345074 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:59.344995 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-gt52v" event={"ID":"c82e3280-90dc-4a6f-858f-66024893e398","Type":"ContainerStarted","Data":"1d06ad70abbff9254b329b41d9038bf405c1001d62e67a99e49dc914f3ad04ac"} Jul 16 21:48:59.346897 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:59.346855 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8d454b4cf-rwfcx" event={"ID":"c3630f3d-ba54-42c2-8fa4-315214cf095a","Type":"ContainerStarted","Data":"76ffd8fc45902f46d72760f2653cf48ac7c8d0bc511480facafd6d9ae9c47715"} Jul 16 21:48:59.423968 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:48:59.423740 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8d454b4cf-rwfcx" podStartSLOduration=7.659895399 podStartE2EDuration="13.423720775s" totalImagesPullingTime="5.763825376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:48:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:59.421428656 +0000 UTC m=+88.043871623" watchObservedRunningTime="2026-07-16 21:48:59.423720775 +0000 UTC m=+88.046163729" Jul 16 21:49:00.354305 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:00.354257 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-gt52v" event={"ID":"c82e3280-90dc-4a6f-858f-66024893e398","Type":"ContainerStarted","Data":"702c8ec4131bb6e34b7c13c3941d199c9aa3d9878cbdafa7eb274a12dc2bcf09"} Jul 16 21:49:00.356113 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:00.356078 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5d897c97d7-c8rk6" event={"ID":"5485377e-20a2-4f1e-b335-8eed2032a4bf","Type":"ContainerStarted","Data":"66111b4bc62cc66464347d13f51647e44eca3cb2d8fe681259402ff6a6f5e577"} Jul 16 21:49:00.358617 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:00.358592 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6ww9h" event={"ID":"4ab63d4c-dc11-4ed8-8f34-4dfcce97f939","Type":"ContainerStarted","Data":"d3692c92093d57424217878ae314c0cf744a13774442d43db8941567b9add2e4"} Jul 16 21:49:00.358734 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:00.358639 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6ww9h" event={"ID":"4ab63d4c-dc11-4ed8-8f34-4dfcce97f939","Type":"ContainerStarted","Data":"a2ee5784a9f86d9f7042371c4b8710511d924c236ddd6015f0f7df8f4980fd42"} Jul 16 21:49:00.360276 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:00.360249 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bbc77df58-9mpjq" event={"ID":"74324078-6d2f-4256-bd15-150bf88c8f0d","Type":"ContainerStarted","Data":"418b0fa854b01a31e37c566d5556d7c92893b1ede221928982a702925a5e905d"} Jul 16 21:49:00.433676 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:00.433624 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-gt52v" podStartSLOduration=7.493010521 podStartE2EDuration="12.433607492s" totalImagesPullingTime="4.940596971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:48:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:49:00.395796599 +0000 UTC m=+89.018239553" watchObservedRunningTime="2026-07-16 21:49:00.433607492 +0000 UTC m=+89.056050444" Jul 16 21:49:00.434567 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:00.434528 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-bbc77df58-9mpjq" podStartSLOduration=3.47940234 podStartE2EDuration="7.434514733s" totalImagesPullingTime="3.955112393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:49:00.431979609 +0000 UTC m=+89.054422562" watchObservedRunningTime="2026-07-16 21:49:00.434514733 +0000 UTC m=+89.056957687" Jul 16 21:49:00.460890 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:00.460829 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6ww9h" podStartSLOduration=7.056368314 podStartE2EDuration="12.460810376s" totalImagesPullingTime="5.274826379s" totalInitContainerRuntime="129.615683ms" isStatefulPod=true podCreationTimestamp="2026-07-16 21:48:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:49:00.458700519 +0000 UTC m=+89.081143472" watchObservedRunningTime="2026-07-16 21:49:00.460810376 +0000 UTC m=+89.083253332" Jul 16 21:49:00.492755 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:00.492660 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5d897c97d7-c8rk6" podStartSLOduration=1.844757547 podStartE2EDuration="7.492637709s" totalImagesPullingTime="5.647880162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:49:00.490028189 +0000 UTC m=+89.112471168" watchObservedRunningTime="2026-07-16 21:49:00.492637709 +0000 UTC m=+89.115080680" Jul 16 21:49:01.337541 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:01.337501 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:49:01.494313 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:01.494267 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:49:02.374051 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:02.374012 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a0419d8-50b3-4681-84a5-500b6c56fcb9","Type":"ContainerStarted","Data":"9c4b9272b99939a9c67d32de72767b3f4e5b600a062f5f4d784b144c6c3f4bb9"} Jul 16 21:49:02.374248 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:02.374081 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a0419d8-50b3-4681-84a5-500b6c56fcb9","Type":"ContainerStarted","Data":"d1c37f49dfb3a9eb1e330b576bf8a2ec91b4de79e4c8141e605278285e3b3981"} Jul 16 21:49:02.374248 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:02.374109 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a0419d8-50b3-4681-84a5-500b6c56fcb9","Type":"ContainerStarted","Data":"d97b6a06fa5520f22745ab2e39c891d6b7a532c47e1aa541b71f4b52dbc27fea"} Jul 16 21:49:02.374248 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:02.374133 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a0419d8-50b3-4681-84a5-500b6c56fcb9","Type":"ContainerStarted","Data":"baea15bd6bfe9014e48bdf34d2605ebb493c9ebce297402f1b01f34c99881399"} Jul 16 21:49:03.197285 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:03.197251 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:49:03.294804 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:03.294765 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:49:03.385023 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:03.384990 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a0419d8-50b3-4681-84a5-500b6c56fcb9","Type":"ContainerStarted","Data":"b28075cfb69540bc6bab08862f3199a80ef334e96440faf50b21d006b7ee0ef1"} Jul 16 21:49:04.139890 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:04.139850 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-bbc77df58-9mpjq" Jul 16 21:49:04.140108 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:04.139911 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-bbc77df58-9mpjq" Jul 16 21:49:04.144825 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:04.144804 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-bbc77df58-9mpjq" Jul 16 21:49:04.392554 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:04.392522 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a0419d8-50b3-4681-84a5-500b6c56fcb9","Type":"ContainerStarted","Data":"afbc9be20357f129858378f0dc51831d3c5917a6604036883229b1deb8ffaae4"} Jul 16 21:49:04.397036 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:04.397009 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-bbc77df58-9mpjq" Jul 16 21:49:04.520052 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:04.520001 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=5.586047875 podStartE2EDuration="14.519983433s" totalImagesPullingTime="8.916264824s" totalInitContainerRuntime="17.670734ms" isStatefulPod=false podCreationTimestamp="2026-07-16 21:48:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:49:04.473944508 +0000 UTC m=+93.096387461" watchObservedRunningTime="2026-07-16 21:49:04.519983433 +0000 UTC m=+93.142426394" Jul 16 21:49:04.870080 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:04.870045 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:49:05.017146 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:05.017111 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:49:07.246156 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:07.246121 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8d454b4cf-rwfcx" Jul 16 21:49:07.681774 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:07.681738 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:49:07.790320 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:07.790288 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:49:13.739129 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:13.739099 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5d897c97d7-c8rk6" Jul 16 21:49:13.739566 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:13.739166 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5d897c97d7-c8rk6" Jul 16 21:49:17.263168 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:17.263136 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zvq8m" Jul 16 21:49:18.180991 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:18.180958 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:49:18.280546 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:18.280514 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:49:24.387487 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:24.387430 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8d454b4cf-rwfcx" podUID="c3630f3d-ba54-42c2-8fa4-315214cf095a" containerName="console" containerID="cri-o://76ffd8fc45902f46d72760f2653cf48ac7c8d0bc511480facafd6d9ae9c47715" gracePeriod=15 Jul 16 21:49:24.654538 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:24.654459 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8d454b4cf-rwfcx" Jul 16 21:49:24.837240 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:24.837191 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c3630f3d-ba54-42c2-8fa4-315214cf095a-service-ca\") pod \"c3630f3d-ba54-42c2-8fa4-315214cf095a\" (UID: \"c3630f3d-ba54-42c2-8fa4-315214cf095a\") " Jul 16 21:49:24.837428 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:24.837253 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c3630f3d-ba54-42c2-8fa4-315214cf095a-console-oauth-config\") pod \"c3630f3d-ba54-42c2-8fa4-315214cf095a\" (UID: \"c3630f3d-ba54-42c2-8fa4-315214cf095a\") " Jul 16 21:49:24.837428 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:24.837276 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3630f3d-ba54-42c2-8fa4-315214cf095a-trusted-ca-bundle\") pod \"c3630f3d-ba54-42c2-8fa4-315214cf095a\" (UID: \"c3630f3d-ba54-42c2-8fa4-315214cf095a\") " Jul 16 21:49:24.837428 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:24.837315 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c3630f3d-ba54-42c2-8fa4-315214cf095a-oauth-serving-cert\") pod \"c3630f3d-ba54-42c2-8fa4-315214cf095a\" (UID: \"c3630f3d-ba54-42c2-8fa4-315214cf095a\") " Jul 16 21:49:24.837428 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:24.837350 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c3630f3d-ba54-42c2-8fa4-315214cf095a-console-config\") pod \"c3630f3d-ba54-42c2-8fa4-315214cf095a\" (UID: \"c3630f3d-ba54-42c2-8fa4-315214cf095a\") " Jul 16 21:49:24.837428 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:24.837374 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c3630f3d-ba54-42c2-8fa4-315214cf095a-console-serving-cert\") pod \"c3630f3d-ba54-42c2-8fa4-315214cf095a\" (UID: \"c3630f3d-ba54-42c2-8fa4-315214cf095a\") " Jul 16 21:49:24.837428 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:24.837407 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cxtn\" (UniqueName: \"kubernetes.io/projected/c3630f3d-ba54-42c2-8fa4-315214cf095a-kube-api-access-9cxtn\") pod \"c3630f3d-ba54-42c2-8fa4-315214cf095a\" (UID: \"c3630f3d-ba54-42c2-8fa4-315214cf095a\") " Jul 16 21:49:24.837739 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:24.837719 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3630f3d-ba54-42c2-8fa4-315214cf095a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c3630f3d-ba54-42c2-8fa4-315214cf095a" (UID: "c3630f3d-ba54-42c2-8fa4-315214cf095a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:49:24.837798 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:24.837732 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3630f3d-ba54-42c2-8fa4-315214cf095a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c3630f3d-ba54-42c2-8fa4-315214cf095a" (UID: "c3630f3d-ba54-42c2-8fa4-315214cf095a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:49:24.837971 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:24.837947 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3630f3d-ba54-42c2-8fa4-315214cf095a-service-ca" (OuterVolumeSpecName: "service-ca") pod "c3630f3d-ba54-42c2-8fa4-315214cf095a" (UID: "c3630f3d-ba54-42c2-8fa4-315214cf095a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:49:24.838045 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:24.837967 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3630f3d-ba54-42c2-8fa4-315214cf095a-console-config" (OuterVolumeSpecName: "console-config") pod "c3630f3d-ba54-42c2-8fa4-315214cf095a" (UID: "c3630f3d-ba54-42c2-8fa4-315214cf095a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:49:24.839654 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:24.839629 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3630f3d-ba54-42c2-8fa4-315214cf095a-kube-api-access-9cxtn" (OuterVolumeSpecName: "kube-api-access-9cxtn") pod "c3630f3d-ba54-42c2-8fa4-315214cf095a" (UID: "c3630f3d-ba54-42c2-8fa4-315214cf095a"). InnerVolumeSpecName "kube-api-access-9cxtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:49:24.839843 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:24.839824 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3630f3d-ba54-42c2-8fa4-315214cf095a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c3630f3d-ba54-42c2-8fa4-315214cf095a" (UID: "c3630f3d-ba54-42c2-8fa4-315214cf095a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:49:24.839905 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:24.839849 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3630f3d-ba54-42c2-8fa4-315214cf095a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c3630f3d-ba54-42c2-8fa4-315214cf095a" (UID: "c3630f3d-ba54-42c2-8fa4-315214cf095a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:49:24.938797 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:24.938727 2585 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c3630f3d-ba54-42c2-8fa4-315214cf095a-oauth-serving-cert\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:49:24.938797 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:24.938752 2585 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c3630f3d-ba54-42c2-8fa4-315214cf095a-console-config\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:49:24.938797 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:24.938762 2585 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c3630f3d-ba54-42c2-8fa4-315214cf095a-console-serving-cert\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:49:24.938797 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:24.938770 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cxtn\" (UniqueName: \"kubernetes.io/projected/c3630f3d-ba54-42c2-8fa4-315214cf095a-kube-api-access-9cxtn\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:49:24.938797 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:24.938779 2585 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c3630f3d-ba54-42c2-8fa4-315214cf095a-service-ca\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:49:24.938797 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:24.938787 2585 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c3630f3d-ba54-42c2-8fa4-315214cf095a-console-oauth-config\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:49:24.938797 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:24.938795 2585 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3630f3d-ba54-42c2-8fa4-315214cf095a-trusted-ca-bundle\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:49:25.475976 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:25.475940 2585 generic.go:356] "Generic (PLEG): container finished" podID="c3630f3d-ba54-42c2-8fa4-315214cf095a" containerID="76ffd8fc45902f46d72760f2653cf48ac7c8d0bc511480facafd6d9ae9c47715" exitCode=0 Jul 16 21:49:25.476454 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:25.475983 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8d454b4cf-rwfcx" Jul 16 21:49:25.476454 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:25.476025 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8d454b4cf-rwfcx" event={"ID":"c3630f3d-ba54-42c2-8fa4-315214cf095a","Type":"ContainerDied","Data":"76ffd8fc45902f46d72760f2653cf48ac7c8d0bc511480facafd6d9ae9c47715"} Jul 16 21:49:25.476454 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:25.476077 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8d454b4cf-rwfcx" event={"ID":"c3630f3d-ba54-42c2-8fa4-315214cf095a","Type":"ContainerDied","Data":"6f085e88365bdf05137c7891f1a25a6c466c02422acc6cc9fa9d9e4ade393194"} Jul 16 21:49:25.476454 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:25.476095 2585 scope.go:117] "RemoveContainer" containerID="76ffd8fc45902f46d72760f2653cf48ac7c8d0bc511480facafd6d9ae9c47715" Jul 16 21:49:25.489268 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:25.489249 2585 scope.go:117] "RemoveContainer" containerID="76ffd8fc45902f46d72760f2653cf48ac7c8d0bc511480facafd6d9ae9c47715" Jul 16 21:49:25.489549 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:49:25.489523 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76ffd8fc45902f46d72760f2653cf48ac7c8d0bc511480facafd6d9ae9c47715\": container with ID starting with 76ffd8fc45902f46d72760f2653cf48ac7c8d0bc511480facafd6d9ae9c47715 not found: ID does not exist" containerID="76ffd8fc45902f46d72760f2653cf48ac7c8d0bc511480facafd6d9ae9c47715" Jul 16 21:49:25.489634 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:25.489555 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76ffd8fc45902f46d72760f2653cf48ac7c8d0bc511480facafd6d9ae9c47715"} err="failed to get container status \"76ffd8fc45902f46d72760f2653cf48ac7c8d0bc511480facafd6d9ae9c47715\": rpc error: code = NotFound desc = could not find container \"76ffd8fc45902f46d72760f2653cf48ac7c8d0bc511480facafd6d9ae9c47715\": container with ID starting with 76ffd8fc45902f46d72760f2653cf48ac7c8d0bc511480facafd6d9ae9c47715 not found: ID does not exist" Jul 16 21:49:25.504661 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:25.504637 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8d454b4cf-rwfcx"] Jul 16 21:49:25.506526 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:25.506508 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8d454b4cf-rwfcx"] Jul 16 21:49:25.933162 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:25.933128 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3630f3d-ba54-42c2-8fa4-315214cf095a" path="/var/lib/kubelet/pods/c3630f3d-ba54-42c2-8fa4-315214cf095a/volumes" Jul 16 21:49:33.744409 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:33.744381 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5d897c97d7-c8rk6" Jul 16 21:49:33.748152 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:49:33.748131 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5d897c97d7-c8rk6" Jul 16 21:50:08.680306 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:08.680265 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:50:08.761337 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:08.761309 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:50:09.684068 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:09.684034 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 21:50:09.684494 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:09.684345 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6a0419d8-50b3-4681-84a5-500b6c56fcb9" containerName="alertmanager" containerID="cri-o://baea15bd6bfe9014e48bdf34d2605ebb493c9ebce297402f1b01f34c99881399" gracePeriod=120 Jul 16 21:50:09.684494 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:09.684408 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6a0419d8-50b3-4681-84a5-500b6c56fcb9" containerName="kube-rbac-proxy-metric" containerID="cri-o://b28075cfb69540bc6bab08862f3199a80ef334e96440faf50b21d006b7ee0ef1" gracePeriod=120 Jul 16 21:50:09.684494 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:09.684432 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6a0419d8-50b3-4681-84a5-500b6c56fcb9" containerName="kube-rbac-proxy-web" containerID="cri-o://d1c37f49dfb3a9eb1e330b576bf8a2ec91b4de79e4c8141e605278285e3b3981" gracePeriod=120 Jul 16 21:50:09.684494 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:09.684478 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6a0419d8-50b3-4681-84a5-500b6c56fcb9" containerName="config-reloader" containerID="cri-o://d97b6a06fa5520f22745ab2e39c891d6b7a532c47e1aa541b71f4b52dbc27fea" gracePeriod=120 Jul 16 21:50:09.684722 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:09.684478 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6a0419d8-50b3-4681-84a5-500b6c56fcb9" containerName="kube-rbac-proxy" containerID="cri-o://9c4b9272b99939a9c67d32de72767b3f4e5b600a062f5f4d784b144c6c3f4bb9" gracePeriod=120 Jul 16 21:50:09.684722 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:09.684491 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6a0419d8-50b3-4681-84a5-500b6c56fcb9" containerName="prom-label-proxy" containerID="cri-o://afbc9be20357f129858378f0dc51831d3c5917a6604036883229b1deb8ffaae4" gracePeriod=120 Jul 16 21:50:10.643048 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:10.643016 2585 generic.go:356] "Generic (PLEG): container finished" podID="6a0419d8-50b3-4681-84a5-500b6c56fcb9" containerID="afbc9be20357f129858378f0dc51831d3c5917a6604036883229b1deb8ffaae4" exitCode=0 Jul 16 21:50:10.643048 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:10.643039 2585 generic.go:356] "Generic (PLEG): container finished" podID="6a0419d8-50b3-4681-84a5-500b6c56fcb9" containerID="9c4b9272b99939a9c67d32de72767b3f4e5b600a062f5f4d784b144c6c3f4bb9" exitCode=0 Jul 16 21:50:10.643048 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:10.643045 2585 generic.go:356] "Generic (PLEG): container finished" podID="6a0419d8-50b3-4681-84a5-500b6c56fcb9" containerID="d97b6a06fa5520f22745ab2e39c891d6b7a532c47e1aa541b71f4b52dbc27fea" exitCode=0 Jul 16 21:50:10.643048 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:10.643051 2585 generic.go:356] "Generic (PLEG): container finished" podID="6a0419d8-50b3-4681-84a5-500b6c56fcb9" containerID="baea15bd6bfe9014e48bdf34d2605ebb493c9ebce297402f1b01f34c99881399" exitCode=0 Jul 16 21:50:10.643322 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:10.643084 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a0419d8-50b3-4681-84a5-500b6c56fcb9","Type":"ContainerDied","Data":"afbc9be20357f129858378f0dc51831d3c5917a6604036883229b1deb8ffaae4"} Jul 16 21:50:10.643322 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:10.643146 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a0419d8-50b3-4681-84a5-500b6c56fcb9","Type":"ContainerDied","Data":"9c4b9272b99939a9c67d32de72767b3f4e5b600a062f5f4d784b144c6c3f4bb9"} Jul 16 21:50:10.643322 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:10.643174 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a0419d8-50b3-4681-84a5-500b6c56fcb9","Type":"ContainerDied","Data":"d97b6a06fa5520f22745ab2e39c891d6b7a532c47e1aa541b71f4b52dbc27fea"} Jul 16 21:50:10.643322 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:10.643199 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a0419d8-50b3-4681-84a5-500b6c56fcb9","Type":"ContainerDied","Data":"baea15bd6bfe9014e48bdf34d2605ebb493c9ebce297402f1b01f34c99881399"} Jul 16 21:50:11.016800 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.016777 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:50:11.103836 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.103807 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6a0419d8-50b3-4681-84a5-500b6c56fcb9-secret-alertmanager-kube-rbac-proxy\") pod \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " Jul 16 21:50:11.104020 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.103852 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6a0419d8-50b3-4681-84a5-500b6c56fcb9-secret-alertmanager-kube-rbac-proxy-web\") pod \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " Jul 16 21:50:11.104020 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.103881 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6a0419d8-50b3-4681-84a5-500b6c56fcb9-secret-alertmanager-main-tls\") pod \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " Jul 16 21:50:11.104020 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.103903 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6z9zw\" (UniqueName: \"kubernetes.io/projected/6a0419d8-50b3-4681-84a5-500b6c56fcb9-kube-api-access-6z9zw\") pod \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " Jul 16 21:50:11.104020 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.103948 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a0419d8-50b3-4681-84a5-500b6c56fcb9-alertmanager-trusted-ca-bundle\") pod \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " Jul 16 21:50:11.104020 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.103975 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a0419d8-50b3-4681-84a5-500b6c56fcb9-metrics-client-ca\") pod \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " Jul 16 21:50:11.104020 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.103996 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6a0419d8-50b3-4681-84a5-500b6c56fcb9-tls-assets\") pod \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " Jul 16 21:50:11.104020 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.104018 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6a0419d8-50b3-4681-84a5-500b6c56fcb9-config-volume\") pod \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " Jul 16 21:50:11.104382 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.104250 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6a0419d8-50b3-4681-84a5-500b6c56fcb9-config-out\") pod \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " Jul 16 21:50:11.104382 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.104307 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6a0419d8-50b3-4681-84a5-500b6c56fcb9-web-config\") pod \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " Jul 16 21:50:11.104382 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.104354 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6a0419d8-50b3-4681-84a5-500b6c56fcb9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " Jul 16 21:50:11.104548 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.104406 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6a0419d8-50b3-4681-84a5-500b6c56fcb9-alertmanager-main-db\") pod \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " Jul 16 21:50:11.104548 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.104451 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6a0419d8-50b3-4681-84a5-500b6c56fcb9-cluster-tls-config\") pod \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\" (UID: \"6a0419d8-50b3-4681-84a5-500b6c56fcb9\") " Jul 16 21:50:11.104548 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.104468 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a0419d8-50b3-4681-84a5-500b6c56fcb9-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "6a0419d8-50b3-4681-84a5-500b6c56fcb9" (UID: "6a0419d8-50b3-4681-84a5-500b6c56fcb9"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:50:11.104548 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.104474 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a0419d8-50b3-4681-84a5-500b6c56fcb9-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "6a0419d8-50b3-4681-84a5-500b6c56fcb9" (UID: "6a0419d8-50b3-4681-84a5-500b6c56fcb9"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:50:11.104762 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.104695 2585 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a0419d8-50b3-4681-84a5-500b6c56fcb9-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:50:11.104762 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.104723 2585 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a0419d8-50b3-4681-84a5-500b6c56fcb9-metrics-client-ca\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:50:11.105394 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.105366 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a0419d8-50b3-4681-84a5-500b6c56fcb9-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "6a0419d8-50b3-4681-84a5-500b6c56fcb9" (UID: "6a0419d8-50b3-4681-84a5-500b6c56fcb9"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:50:11.106584 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.106555 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a0419d8-50b3-4681-84a5-500b6c56fcb9-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "6a0419d8-50b3-4681-84a5-500b6c56fcb9" (UID: "6a0419d8-50b3-4681-84a5-500b6c56fcb9"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:50:11.106701 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.106608 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a0419d8-50b3-4681-84a5-500b6c56fcb9-config-volume" (OuterVolumeSpecName: "config-volume") pod "6a0419d8-50b3-4681-84a5-500b6c56fcb9" (UID: "6a0419d8-50b3-4681-84a5-500b6c56fcb9"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:50:11.107016 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.106978 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a0419d8-50b3-4681-84a5-500b6c56fcb9-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "6a0419d8-50b3-4681-84a5-500b6c56fcb9" (UID: "6a0419d8-50b3-4681-84a5-500b6c56fcb9"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:50:11.107016 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.106994 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a0419d8-50b3-4681-84a5-500b6c56fcb9-kube-api-access-6z9zw" (OuterVolumeSpecName: "kube-api-access-6z9zw") pod "6a0419d8-50b3-4681-84a5-500b6c56fcb9" (UID: "6a0419d8-50b3-4681-84a5-500b6c56fcb9"). InnerVolumeSpecName "kube-api-access-6z9zw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:50:11.107650 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.107626 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a0419d8-50b3-4681-84a5-500b6c56fcb9-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "6a0419d8-50b3-4681-84a5-500b6c56fcb9" (UID: "6a0419d8-50b3-4681-84a5-500b6c56fcb9"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:50:11.108071 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.108054 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a0419d8-50b3-4681-84a5-500b6c56fcb9-config-out" (OuterVolumeSpecName: "config-out") pod "6a0419d8-50b3-4681-84a5-500b6c56fcb9" (UID: "6a0419d8-50b3-4681-84a5-500b6c56fcb9"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:50:11.108135 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.108072 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a0419d8-50b3-4681-84a5-500b6c56fcb9-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "6a0419d8-50b3-4681-84a5-500b6c56fcb9" (UID: "6a0419d8-50b3-4681-84a5-500b6c56fcb9"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:50:11.108249 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.108200 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a0419d8-50b3-4681-84a5-500b6c56fcb9-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "6a0419d8-50b3-4681-84a5-500b6c56fcb9" (UID: "6a0419d8-50b3-4681-84a5-500b6c56fcb9"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:50:11.111802 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.111743 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a0419d8-50b3-4681-84a5-500b6c56fcb9-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "6a0419d8-50b3-4681-84a5-500b6c56fcb9" (UID: "6a0419d8-50b3-4681-84a5-500b6c56fcb9"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:50:11.116961 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.116936 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a0419d8-50b3-4681-84a5-500b6c56fcb9-web-config" (OuterVolumeSpecName: "web-config") pod "6a0419d8-50b3-4681-84a5-500b6c56fcb9" (UID: "6a0419d8-50b3-4681-84a5-500b6c56fcb9"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:50:11.205774 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.205711 2585 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6a0419d8-50b3-4681-84a5-500b6c56fcb9-tls-assets\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:50:11.205774 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.205737 2585 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6a0419d8-50b3-4681-84a5-500b6c56fcb9-config-volume\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:50:11.205774 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.205746 2585 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6a0419d8-50b3-4681-84a5-500b6c56fcb9-config-out\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:50:11.205774 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.205754 2585 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6a0419d8-50b3-4681-84a5-500b6c56fcb9-web-config\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:50:11.205774 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.205763 2585 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6a0419d8-50b3-4681-84a5-500b6c56fcb9-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:50:11.205774 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.205771 2585 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6a0419d8-50b3-4681-84a5-500b6c56fcb9-alertmanager-main-db\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:50:11.206044 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.205780 2585 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6a0419d8-50b3-4681-84a5-500b6c56fcb9-cluster-tls-config\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:50:11.206044 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.205789 2585 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6a0419d8-50b3-4681-84a5-500b6c56fcb9-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:50:11.206044 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.205798 2585 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6a0419d8-50b3-4681-84a5-500b6c56fcb9-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:50:11.206044 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.205808 2585 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6a0419d8-50b3-4681-84a5-500b6c56fcb9-secret-alertmanager-main-tls\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:50:11.206044 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.205819 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z9zw\" (UniqueName: \"kubernetes.io/projected/6a0419d8-50b3-4681-84a5-500b6c56fcb9-kube-api-access-6z9zw\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:50:11.650234 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.650192 2585 generic.go:356] "Generic (PLEG): container finished" podID="6a0419d8-50b3-4681-84a5-500b6c56fcb9" containerID="b28075cfb69540bc6bab08862f3199a80ef334e96440faf50b21d006b7ee0ef1" exitCode=0 Jul 16 21:50:11.650234 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.650237 2585 generic.go:356] "Generic (PLEG): container finished" podID="6a0419d8-50b3-4681-84a5-500b6c56fcb9" containerID="d1c37f49dfb3a9eb1e330b576bf8a2ec91b4de79e4c8141e605278285e3b3981" exitCode=0 Jul 16 21:50:11.650404 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.650266 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a0419d8-50b3-4681-84a5-500b6c56fcb9","Type":"ContainerDied","Data":"b28075cfb69540bc6bab08862f3199a80ef334e96440faf50b21d006b7ee0ef1"} Jul 16 21:50:11.650404 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.650294 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:50:11.650404 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.650337 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a0419d8-50b3-4681-84a5-500b6c56fcb9","Type":"ContainerDied","Data":"d1c37f49dfb3a9eb1e330b576bf8a2ec91b4de79e4c8141e605278285e3b3981"} Jul 16 21:50:11.650404 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.650365 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a0419d8-50b3-4681-84a5-500b6c56fcb9","Type":"ContainerDied","Data":"b56c1f01e5a3b1aea5bb2e04ed8b42c05c1149bc5e079bc20eeefadf218f4bbf"} Jul 16 21:50:11.650404 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.650394 2585 scope.go:117] "RemoveContainer" containerID="afbc9be20357f129858378f0dc51831d3c5917a6604036883229b1deb8ffaae4" Jul 16 21:50:11.660008 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.659988 2585 scope.go:117] "RemoveContainer" containerID="b28075cfb69540bc6bab08862f3199a80ef334e96440faf50b21d006b7ee0ef1" Jul 16 21:50:11.670078 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.670061 2585 scope.go:117] "RemoveContainer" containerID="9c4b9272b99939a9c67d32de72767b3f4e5b600a062f5f4d784b144c6c3f4bb9" Jul 16 21:50:11.676165 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.676148 2585 scope.go:117] "RemoveContainer" containerID="d1c37f49dfb3a9eb1e330b576bf8a2ec91b4de79e4c8141e605278285e3b3981" Jul 16 21:50:11.680493 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.680474 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 21:50:11.682857 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.682836 2585 scope.go:117] "RemoveContainer" containerID="d97b6a06fa5520f22745ab2e39c891d6b7a532c47e1aa541b71f4b52dbc27fea" Jul 16 21:50:11.687341 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.687322 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 21:50:11.689654 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.689637 2585 scope.go:117] "RemoveContainer" containerID="baea15bd6bfe9014e48bdf34d2605ebb493c9ebce297402f1b01f34c99881399" Jul 16 21:50:11.696121 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.696103 2585 scope.go:117] "RemoveContainer" containerID="470ef4afef9b1a707ee8f28b470232db59e90b78df88c405cdd8ad6e74a98046" Jul 16 21:50:11.702138 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.702124 2585 scope.go:117] "RemoveContainer" containerID="afbc9be20357f129858378f0dc51831d3c5917a6604036883229b1deb8ffaae4" Jul 16 21:50:11.702410 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:50:11.702389 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afbc9be20357f129858378f0dc51831d3c5917a6604036883229b1deb8ffaae4\": container with ID starting with afbc9be20357f129858378f0dc51831d3c5917a6604036883229b1deb8ffaae4 not found: ID does not exist" containerID="afbc9be20357f129858378f0dc51831d3c5917a6604036883229b1deb8ffaae4" Jul 16 21:50:11.702483 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.702417 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afbc9be20357f129858378f0dc51831d3c5917a6604036883229b1deb8ffaae4"} err="failed to get container status \"afbc9be20357f129858378f0dc51831d3c5917a6604036883229b1deb8ffaae4\": rpc error: code = NotFound desc = could not find container \"afbc9be20357f129858378f0dc51831d3c5917a6604036883229b1deb8ffaae4\": container with ID starting with afbc9be20357f129858378f0dc51831d3c5917a6604036883229b1deb8ffaae4 not found: ID does not exist" Jul 16 21:50:11.702483 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.702435 2585 scope.go:117] "RemoveContainer" containerID="b28075cfb69540bc6bab08862f3199a80ef334e96440faf50b21d006b7ee0ef1" Jul 16 21:50:11.702674 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:50:11.702657 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b28075cfb69540bc6bab08862f3199a80ef334e96440faf50b21d006b7ee0ef1\": container with ID starting with b28075cfb69540bc6bab08862f3199a80ef334e96440faf50b21d006b7ee0ef1 not found: ID does not exist" containerID="b28075cfb69540bc6bab08862f3199a80ef334e96440faf50b21d006b7ee0ef1" Jul 16 21:50:11.702713 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.702680 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b28075cfb69540bc6bab08862f3199a80ef334e96440faf50b21d006b7ee0ef1"} err="failed to get container status \"b28075cfb69540bc6bab08862f3199a80ef334e96440faf50b21d006b7ee0ef1\": rpc error: code = NotFound desc = could not find container \"b28075cfb69540bc6bab08862f3199a80ef334e96440faf50b21d006b7ee0ef1\": container with ID starting with b28075cfb69540bc6bab08862f3199a80ef334e96440faf50b21d006b7ee0ef1 not found: ID does not exist" Jul 16 21:50:11.702713 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.702695 2585 scope.go:117] "RemoveContainer" containerID="9c4b9272b99939a9c67d32de72767b3f4e5b600a062f5f4d784b144c6c3f4bb9" Jul 16 21:50:11.702882 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:50:11.702869 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c4b9272b99939a9c67d32de72767b3f4e5b600a062f5f4d784b144c6c3f4bb9\": container with ID starting with 9c4b9272b99939a9c67d32de72767b3f4e5b600a062f5f4d784b144c6c3f4bb9 not found: ID does not exist" containerID="9c4b9272b99939a9c67d32de72767b3f4e5b600a062f5f4d784b144c6c3f4bb9" Jul 16 21:50:11.702923 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.702884 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c4b9272b99939a9c67d32de72767b3f4e5b600a062f5f4d784b144c6c3f4bb9"} err="failed to get container status \"9c4b9272b99939a9c67d32de72767b3f4e5b600a062f5f4d784b144c6c3f4bb9\": rpc error: code = NotFound desc = could not find container \"9c4b9272b99939a9c67d32de72767b3f4e5b600a062f5f4d784b144c6c3f4bb9\": container with ID starting with 9c4b9272b99939a9c67d32de72767b3f4e5b600a062f5f4d784b144c6c3f4bb9 not found: ID does not exist" Jul 16 21:50:11.702923 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.702896 2585 scope.go:117] "RemoveContainer" containerID="d1c37f49dfb3a9eb1e330b576bf8a2ec91b4de79e4c8141e605278285e3b3981" Jul 16 21:50:11.703110 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:50:11.703091 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1c37f49dfb3a9eb1e330b576bf8a2ec91b4de79e4c8141e605278285e3b3981\": container with ID starting with d1c37f49dfb3a9eb1e330b576bf8a2ec91b4de79e4c8141e605278285e3b3981 not found: ID does not exist" containerID="d1c37f49dfb3a9eb1e330b576bf8a2ec91b4de79e4c8141e605278285e3b3981" Jul 16 21:50:11.703181 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.703118 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1c37f49dfb3a9eb1e330b576bf8a2ec91b4de79e4c8141e605278285e3b3981"} err="failed to get container status \"d1c37f49dfb3a9eb1e330b576bf8a2ec91b4de79e4c8141e605278285e3b3981\": rpc error: code = NotFound desc = could not find container \"d1c37f49dfb3a9eb1e330b576bf8a2ec91b4de79e4c8141e605278285e3b3981\": container with ID starting with d1c37f49dfb3a9eb1e330b576bf8a2ec91b4de79e4c8141e605278285e3b3981 not found: ID does not exist" Jul 16 21:50:11.703181 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.703140 2585 scope.go:117] "RemoveContainer" containerID="d97b6a06fa5520f22745ab2e39c891d6b7a532c47e1aa541b71f4b52dbc27fea" Jul 16 21:50:11.703421 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:50:11.703404 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d97b6a06fa5520f22745ab2e39c891d6b7a532c47e1aa541b71f4b52dbc27fea\": container with ID starting with d97b6a06fa5520f22745ab2e39c891d6b7a532c47e1aa541b71f4b52dbc27fea not found: ID does not exist" containerID="d97b6a06fa5520f22745ab2e39c891d6b7a532c47e1aa541b71f4b52dbc27fea" Jul 16 21:50:11.703474 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.703425 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d97b6a06fa5520f22745ab2e39c891d6b7a532c47e1aa541b71f4b52dbc27fea"} err="failed to get container status \"d97b6a06fa5520f22745ab2e39c891d6b7a532c47e1aa541b71f4b52dbc27fea\": rpc error: code = NotFound desc = could not find container \"d97b6a06fa5520f22745ab2e39c891d6b7a532c47e1aa541b71f4b52dbc27fea\": container with ID starting with d97b6a06fa5520f22745ab2e39c891d6b7a532c47e1aa541b71f4b52dbc27fea not found: ID does not exist" Jul 16 21:50:11.703474 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.703438 2585 scope.go:117] "RemoveContainer" containerID="baea15bd6bfe9014e48bdf34d2605ebb493c9ebce297402f1b01f34c99881399" Jul 16 21:50:11.703687 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:50:11.703673 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baea15bd6bfe9014e48bdf34d2605ebb493c9ebce297402f1b01f34c99881399\": container with ID starting with baea15bd6bfe9014e48bdf34d2605ebb493c9ebce297402f1b01f34c99881399 not found: ID does not exist" containerID="baea15bd6bfe9014e48bdf34d2605ebb493c9ebce297402f1b01f34c99881399" Jul 16 21:50:11.703736 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.703690 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baea15bd6bfe9014e48bdf34d2605ebb493c9ebce297402f1b01f34c99881399"} err="failed to get container status \"baea15bd6bfe9014e48bdf34d2605ebb493c9ebce297402f1b01f34c99881399\": rpc error: code = NotFound desc = could not find container \"baea15bd6bfe9014e48bdf34d2605ebb493c9ebce297402f1b01f34c99881399\": container with ID starting with baea15bd6bfe9014e48bdf34d2605ebb493c9ebce297402f1b01f34c99881399 not found: ID does not exist" Jul 16 21:50:11.703736 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.703702 2585 scope.go:117] "RemoveContainer" containerID="470ef4afef9b1a707ee8f28b470232db59e90b78df88c405cdd8ad6e74a98046" Jul 16 21:50:11.703905 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:50:11.703891 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"470ef4afef9b1a707ee8f28b470232db59e90b78df88c405cdd8ad6e74a98046\": container with ID starting with 470ef4afef9b1a707ee8f28b470232db59e90b78df88c405cdd8ad6e74a98046 not found: ID does not exist" containerID="470ef4afef9b1a707ee8f28b470232db59e90b78df88c405cdd8ad6e74a98046" Jul 16 21:50:11.703946 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.703908 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"470ef4afef9b1a707ee8f28b470232db59e90b78df88c405cdd8ad6e74a98046"} err="failed to get container status \"470ef4afef9b1a707ee8f28b470232db59e90b78df88c405cdd8ad6e74a98046\": rpc error: code = NotFound desc = could not find container \"470ef4afef9b1a707ee8f28b470232db59e90b78df88c405cdd8ad6e74a98046\": container with ID starting with 470ef4afef9b1a707ee8f28b470232db59e90b78df88c405cdd8ad6e74a98046 not found: ID does not exist" Jul 16 21:50:11.703946 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.703919 2585 scope.go:117] "RemoveContainer" containerID="afbc9be20357f129858378f0dc51831d3c5917a6604036883229b1deb8ffaae4" Jul 16 21:50:11.704136 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.704115 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afbc9be20357f129858378f0dc51831d3c5917a6604036883229b1deb8ffaae4"} err="failed to get container status \"afbc9be20357f129858378f0dc51831d3c5917a6604036883229b1deb8ffaae4\": rpc error: code = NotFound desc = could not find container \"afbc9be20357f129858378f0dc51831d3c5917a6604036883229b1deb8ffaae4\": container with ID starting with afbc9be20357f129858378f0dc51831d3c5917a6604036883229b1deb8ffaae4 not found: ID does not exist" Jul 16 21:50:11.704136 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.704136 2585 scope.go:117] "RemoveContainer" containerID="b28075cfb69540bc6bab08862f3199a80ef334e96440faf50b21d006b7ee0ef1" Jul 16 21:50:11.704376 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.704358 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b28075cfb69540bc6bab08862f3199a80ef334e96440faf50b21d006b7ee0ef1"} err="failed to get container status \"b28075cfb69540bc6bab08862f3199a80ef334e96440faf50b21d006b7ee0ef1\": rpc error: code = NotFound desc = could not find container \"b28075cfb69540bc6bab08862f3199a80ef334e96440faf50b21d006b7ee0ef1\": container with ID starting with b28075cfb69540bc6bab08862f3199a80ef334e96440faf50b21d006b7ee0ef1 not found: ID does not exist" Jul 16 21:50:11.704376 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.704377 2585 scope.go:117] "RemoveContainer" containerID="9c4b9272b99939a9c67d32de72767b3f4e5b600a062f5f4d784b144c6c3f4bb9" Jul 16 21:50:11.704571 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.704550 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c4b9272b99939a9c67d32de72767b3f4e5b600a062f5f4d784b144c6c3f4bb9"} err="failed to get container status \"9c4b9272b99939a9c67d32de72767b3f4e5b600a062f5f4d784b144c6c3f4bb9\": rpc error: code = NotFound desc = could not find container \"9c4b9272b99939a9c67d32de72767b3f4e5b600a062f5f4d784b144c6c3f4bb9\": container with ID starting with 9c4b9272b99939a9c67d32de72767b3f4e5b600a062f5f4d784b144c6c3f4bb9 not found: ID does not exist" Jul 16 21:50:11.704622 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.704571 2585 scope.go:117] "RemoveContainer" containerID="d1c37f49dfb3a9eb1e330b576bf8a2ec91b4de79e4c8141e605278285e3b3981" Jul 16 21:50:11.704752 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.704736 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1c37f49dfb3a9eb1e330b576bf8a2ec91b4de79e4c8141e605278285e3b3981"} err="failed to get container status \"d1c37f49dfb3a9eb1e330b576bf8a2ec91b4de79e4c8141e605278285e3b3981\": rpc error: code = NotFound desc = could not find container \"d1c37f49dfb3a9eb1e330b576bf8a2ec91b4de79e4c8141e605278285e3b3981\": container with ID starting with d1c37f49dfb3a9eb1e330b576bf8a2ec91b4de79e4c8141e605278285e3b3981 not found: ID does not exist" Jul 16 21:50:11.704799 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.704751 2585 scope.go:117] "RemoveContainer" containerID="d97b6a06fa5520f22745ab2e39c891d6b7a532c47e1aa541b71f4b52dbc27fea" Jul 16 21:50:11.704932 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.704919 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d97b6a06fa5520f22745ab2e39c891d6b7a532c47e1aa541b71f4b52dbc27fea"} err="failed to get container status \"d97b6a06fa5520f22745ab2e39c891d6b7a532c47e1aa541b71f4b52dbc27fea\": rpc error: code = NotFound desc = could not find container \"d97b6a06fa5520f22745ab2e39c891d6b7a532c47e1aa541b71f4b52dbc27fea\": container with ID starting with d97b6a06fa5520f22745ab2e39c891d6b7a532c47e1aa541b71f4b52dbc27fea not found: ID does not exist" Jul 16 21:50:11.704932 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.704932 2585 scope.go:117] "RemoveContainer" containerID="baea15bd6bfe9014e48bdf34d2605ebb493c9ebce297402f1b01f34c99881399" Jul 16 21:50:11.705127 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.705112 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baea15bd6bfe9014e48bdf34d2605ebb493c9ebce297402f1b01f34c99881399"} err="failed to get container status \"baea15bd6bfe9014e48bdf34d2605ebb493c9ebce297402f1b01f34c99881399\": rpc error: code = NotFound desc = could not find container \"baea15bd6bfe9014e48bdf34d2605ebb493c9ebce297402f1b01f34c99881399\": container with ID starting with baea15bd6bfe9014e48bdf34d2605ebb493c9ebce297402f1b01f34c99881399 not found: ID does not exist" Jul 16 21:50:11.705162 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.705127 2585 scope.go:117] "RemoveContainer" containerID="470ef4afef9b1a707ee8f28b470232db59e90b78df88c405cdd8ad6e74a98046" Jul 16 21:50:11.705317 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.705302 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"470ef4afef9b1a707ee8f28b470232db59e90b78df88c405cdd8ad6e74a98046"} err="failed to get container status \"470ef4afef9b1a707ee8f28b470232db59e90b78df88c405cdd8ad6e74a98046\": rpc error: code = NotFound desc = could not find container \"470ef4afef9b1a707ee8f28b470232db59e90b78df88c405cdd8ad6e74a98046\": container with ID starting with 470ef4afef9b1a707ee8f28b470232db59e90b78df88c405cdd8ad6e74a98046 not found: ID does not exist" Jul 16 21:50:11.721609 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.721589 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 21:50:11.721875 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.721862 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a0419d8-50b3-4681-84a5-500b6c56fcb9" containerName="init-config-reloader" Jul 16 21:50:11.721925 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.721876 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a0419d8-50b3-4681-84a5-500b6c56fcb9" containerName="init-config-reloader" Jul 16 21:50:11.721925 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.721885 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a0419d8-50b3-4681-84a5-500b6c56fcb9" containerName="alertmanager" Jul 16 21:50:11.721925 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.721890 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a0419d8-50b3-4681-84a5-500b6c56fcb9" containerName="alertmanager" Jul 16 21:50:11.721925 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.721897 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a0419d8-50b3-4681-84a5-500b6c56fcb9" containerName="kube-rbac-proxy" Jul 16 21:50:11.721925 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.721903 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a0419d8-50b3-4681-84a5-500b6c56fcb9" containerName="kube-rbac-proxy" Jul 16 21:50:11.721925 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.721914 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a0419d8-50b3-4681-84a5-500b6c56fcb9" containerName="prom-label-proxy" Jul 16 21:50:11.721925 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.721919 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a0419d8-50b3-4681-84a5-500b6c56fcb9" containerName="prom-label-proxy" Jul 16 21:50:11.721925 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.721927 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a0419d8-50b3-4681-84a5-500b6c56fcb9" containerName="config-reloader" Jul 16 21:50:11.722197 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.721933 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a0419d8-50b3-4681-84a5-500b6c56fcb9" containerName="config-reloader" Jul 16 21:50:11.722197 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.721939 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a0419d8-50b3-4681-84a5-500b6c56fcb9" containerName="kube-rbac-proxy-metric" Jul 16 21:50:11.722197 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.721944 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a0419d8-50b3-4681-84a5-500b6c56fcb9" containerName="kube-rbac-proxy-metric" Jul 16 21:50:11.722197 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.721953 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a0419d8-50b3-4681-84a5-500b6c56fcb9" containerName="kube-rbac-proxy-web" Jul 16 21:50:11.722197 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.721958 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a0419d8-50b3-4681-84a5-500b6c56fcb9" containerName="kube-rbac-proxy-web" Jul 16 21:50:11.722197 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.721965 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3630f3d-ba54-42c2-8fa4-315214cf095a" containerName="console" Jul 16 21:50:11.722197 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.721970 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3630f3d-ba54-42c2-8fa4-315214cf095a" containerName="console" Jul 16 21:50:11.722197 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.722017 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a0419d8-50b3-4681-84a5-500b6c56fcb9" containerName="alertmanager" Jul 16 21:50:11.722197 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.722047 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3630f3d-ba54-42c2-8fa4-315214cf095a" containerName="console" Jul 16 21:50:11.722197 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.722053 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a0419d8-50b3-4681-84a5-500b6c56fcb9" containerName="kube-rbac-proxy-web" Jul 16 21:50:11.722197 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.722059 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a0419d8-50b3-4681-84a5-500b6c56fcb9" containerName="kube-rbac-proxy" Jul 16 21:50:11.722197 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.722065 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a0419d8-50b3-4681-84a5-500b6c56fcb9" containerName="kube-rbac-proxy-metric" Jul 16 21:50:11.722197 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.722071 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a0419d8-50b3-4681-84a5-500b6c56fcb9" containerName="prom-label-proxy" Jul 16 21:50:11.722197 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.722077 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a0419d8-50b3-4681-84a5-500b6c56fcb9" containerName="config-reloader" Jul 16 21:50:11.725593 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.725579 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:50:11.730765 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.730746 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 16 21:50:11.730862 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.730777 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 16 21:50:11.730862 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.730827 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-5f4vw\"" Jul 16 21:50:11.734517 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.734501 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 16 21:50:11.734602 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.734564 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 16 21:50:11.734602 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.734579 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 16 21:50:11.734898 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.734881 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 16 21:50:11.734964 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.734902 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 16 21:50:11.734964 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.734949 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 16 21:50:11.739931 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.739914 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 16 21:50:11.745545 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.745525 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 21:50:11.811201 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.811174 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9fcf5399-29b4-46a1-84ed-2779ac199ead-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9fcf5399-29b4-46a1-84ed-2779ac199ead\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:50:11.811358 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.811209 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fcf5399-29b4-46a1-84ed-2779ac199ead-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9fcf5399-29b4-46a1-84ed-2779ac199ead\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:50:11.811358 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.811251 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9fcf5399-29b4-46a1-84ed-2779ac199ead-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9fcf5399-29b4-46a1-84ed-2779ac199ead\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:50:11.811358 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.811267 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9fcf5399-29b4-46a1-84ed-2779ac199ead-config-out\") pod \"alertmanager-main-0\" (UID: \"9fcf5399-29b4-46a1-84ed-2779ac199ead\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:50:11.811358 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.811282 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9fcf5399-29b4-46a1-84ed-2779ac199ead-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9fcf5399-29b4-46a1-84ed-2779ac199ead\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:50:11.811358 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.811301 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9fcf5399-29b4-46a1-84ed-2779ac199ead-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9fcf5399-29b4-46a1-84ed-2779ac199ead\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:50:11.811358 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.811320 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fcf5399-29b4-46a1-84ed-2779ac199ead-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9fcf5399-29b4-46a1-84ed-2779ac199ead\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:50:11.811589 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.811371 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9fcf5399-29b4-46a1-84ed-2779ac199ead-web-config\") pod \"alertmanager-main-0\" (UID: \"9fcf5399-29b4-46a1-84ed-2779ac199ead\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:50:11.811589 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.811431 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz2q6\" (UniqueName: \"kubernetes.io/projected/9fcf5399-29b4-46a1-84ed-2779ac199ead-kube-api-access-vz2q6\") pod \"alertmanager-main-0\" (UID: \"9fcf5399-29b4-46a1-84ed-2779ac199ead\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:50:11.811589 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.811451 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9fcf5399-29b4-46a1-84ed-2779ac199ead-config-volume\") pod \"alertmanager-main-0\" (UID: \"9fcf5399-29b4-46a1-84ed-2779ac199ead\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:50:11.811589 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.811473 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9fcf5399-29b4-46a1-84ed-2779ac199ead-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9fcf5399-29b4-46a1-84ed-2779ac199ead\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:50:11.811589 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.811491 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9fcf5399-29b4-46a1-84ed-2779ac199ead-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9fcf5399-29b4-46a1-84ed-2779ac199ead\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:50:11.811589 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.811576 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9fcf5399-29b4-46a1-84ed-2779ac199ead-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9fcf5399-29b4-46a1-84ed-2779ac199ead\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:50:11.912582 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.912506 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz2q6\" (UniqueName: \"kubernetes.io/projected/9fcf5399-29b4-46a1-84ed-2779ac199ead-kube-api-access-vz2q6\") pod \"alertmanager-main-0\" (UID: \"9fcf5399-29b4-46a1-84ed-2779ac199ead\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:50:11.912582 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.912534 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9fcf5399-29b4-46a1-84ed-2779ac199ead-config-volume\") pod \"alertmanager-main-0\" (UID: \"9fcf5399-29b4-46a1-84ed-2779ac199ead\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:50:11.912582 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.912555 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9fcf5399-29b4-46a1-84ed-2779ac199ead-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9fcf5399-29b4-46a1-84ed-2779ac199ead\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:50:11.912815 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.912585 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9fcf5399-29b4-46a1-84ed-2779ac199ead-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9fcf5399-29b4-46a1-84ed-2779ac199ead\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:50:11.912815 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.912637 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9fcf5399-29b4-46a1-84ed-2779ac199ead-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9fcf5399-29b4-46a1-84ed-2779ac199ead\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:50:11.912815 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.912678 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9fcf5399-29b4-46a1-84ed-2779ac199ead-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9fcf5399-29b4-46a1-84ed-2779ac199ead\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:50:11.912815 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.912801 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fcf5399-29b4-46a1-84ed-2779ac199ead-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9fcf5399-29b4-46a1-84ed-2779ac199ead\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:50:11.913004 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.912845 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9fcf5399-29b4-46a1-84ed-2779ac199ead-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9fcf5399-29b4-46a1-84ed-2779ac199ead\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:50:11.913004 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.912863 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9fcf5399-29b4-46a1-84ed-2779ac199ead-config-out\") pod \"alertmanager-main-0\" (UID: \"9fcf5399-29b4-46a1-84ed-2779ac199ead\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:50:11.913004 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.912886 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9fcf5399-29b4-46a1-84ed-2779ac199ead-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9fcf5399-29b4-46a1-84ed-2779ac199ead\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:50:11.913004 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.912918 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9fcf5399-29b4-46a1-84ed-2779ac199ead-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9fcf5399-29b4-46a1-84ed-2779ac199ead\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:50:11.913004 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.912946 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fcf5399-29b4-46a1-84ed-2779ac199ead-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9fcf5399-29b4-46a1-84ed-2779ac199ead\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:50:11.913004 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.912972 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9fcf5399-29b4-46a1-84ed-2779ac199ead-web-config\") pod \"alertmanager-main-0\" (UID: \"9fcf5399-29b4-46a1-84ed-2779ac199ead\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:50:11.913004 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.912976 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9fcf5399-29b4-46a1-84ed-2779ac199ead-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9fcf5399-29b4-46a1-84ed-2779ac199ead\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:50:11.913566 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.913544 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fcf5399-29b4-46a1-84ed-2779ac199ead-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9fcf5399-29b4-46a1-84ed-2779ac199ead\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:50:11.913952 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.913909 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fcf5399-29b4-46a1-84ed-2779ac199ead-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9fcf5399-29b4-46a1-84ed-2779ac199ead\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:50:11.915685 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.915659 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9fcf5399-29b4-46a1-84ed-2779ac199ead-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9fcf5399-29b4-46a1-84ed-2779ac199ead\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:50:11.915843 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.915817 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9fcf5399-29b4-46a1-84ed-2779ac199ead-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9fcf5399-29b4-46a1-84ed-2779ac199ead\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:50:11.916114 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.916063 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9fcf5399-29b4-46a1-84ed-2779ac199ead-config-volume\") pod \"alertmanager-main-0\" (UID: \"9fcf5399-29b4-46a1-84ed-2779ac199ead\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:50:11.916211 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.916198 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9fcf5399-29b4-46a1-84ed-2779ac199ead-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9fcf5399-29b4-46a1-84ed-2779ac199ead\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:50:11.916378 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.916357 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9fcf5399-29b4-46a1-84ed-2779ac199ead-config-out\") pod \"alertmanager-main-0\" (UID: \"9fcf5399-29b4-46a1-84ed-2779ac199ead\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:50:11.916463 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.916444 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9fcf5399-29b4-46a1-84ed-2779ac199ead-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9fcf5399-29b4-46a1-84ed-2779ac199ead\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:50:11.916506 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.916464 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9fcf5399-29b4-46a1-84ed-2779ac199ead-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9fcf5399-29b4-46a1-84ed-2779ac199ead\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:50:11.916805 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.916787 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9fcf5399-29b4-46a1-84ed-2779ac199ead-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9fcf5399-29b4-46a1-84ed-2779ac199ead\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:50:11.917150 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.917135 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9fcf5399-29b4-46a1-84ed-2779ac199ead-web-config\") pod \"alertmanager-main-0\" (UID: \"9fcf5399-29b4-46a1-84ed-2779ac199ead\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:50:11.927941 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.927921 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz2q6\" (UniqueName: \"kubernetes.io/projected/9fcf5399-29b4-46a1-84ed-2779ac199ead-kube-api-access-vz2q6\") pod \"alertmanager-main-0\" (UID: \"9fcf5399-29b4-46a1-84ed-2779ac199ead\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:50:11.934109 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:11.934086 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a0419d8-50b3-4681-84a5-500b6c56fcb9" path="/var/lib/kubelet/pods/6a0419d8-50b3-4681-84a5-500b6c56fcb9/volumes" Jul 16 21:50:12.033869 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:12.033841 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:50:12.193114 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:12.193029 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 21:50:12.197440 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:50:12.197413 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fcf5399_29b4_46a1_84ed_2779ac199ead.slice/crio-9731135824dc9d40c8eb3d40764bc35920b03b6eacf9d993c175f13fa1cc1997 WatchSource:0}: Error finding container 9731135824dc9d40c8eb3d40764bc35920b03b6eacf9d993c175f13fa1cc1997: Status 404 returned error can't find the container with id 9731135824dc9d40c8eb3d40764bc35920b03b6eacf9d993c175f13fa1cc1997 Jul 16 21:50:12.660102 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:12.660066 2585 generic.go:356] "Generic (PLEG): container finished" podID="9fcf5399-29b4-46a1-84ed-2779ac199ead" containerID="f4f9af8d06fcea59d1aae24d281a79dd33c82ceab187cbb7f81676c23667c065" exitCode=0 Jul 16 21:50:12.660298 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:12.660118 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9fcf5399-29b4-46a1-84ed-2779ac199ead","Type":"ContainerDied","Data":"f4f9af8d06fcea59d1aae24d281a79dd33c82ceab187cbb7f81676c23667c065"} Jul 16 21:50:12.660298 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:12.660211 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9fcf5399-29b4-46a1-84ed-2779ac199ead","Type":"ContainerStarted","Data":"9731135824dc9d40c8eb3d40764bc35920b03b6eacf9d993c175f13fa1cc1997"} Jul 16 21:50:13.667205 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:13.667169 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9fcf5399-29b4-46a1-84ed-2779ac199ead","Type":"ContainerStarted","Data":"b37d4bdb5c70011dcfa66114510d71c2a699aa9609ad0460713886b4f6615e24"} Jul 16 21:50:13.667591 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:13.667272 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9fcf5399-29b4-46a1-84ed-2779ac199ead","Type":"ContainerStarted","Data":"c9983a90dce485eba959fd6e040829f032f947f11e33b26dc3e256a9b8340479"} Jul 16 21:50:13.667591 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:13.667316 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9fcf5399-29b4-46a1-84ed-2779ac199ead","Type":"ContainerStarted","Data":"fdb6c70d56d5f99a52bcd0b61910200beb11719cfd19162f59bc470959c55286"} Jul 16 21:50:13.667591 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:13.667360 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9fcf5399-29b4-46a1-84ed-2779ac199ead","Type":"ContainerStarted","Data":"f04f819cbfeed6b63b53de7c77605d1e8951b38b9fe46311cc835c9f08368f15"} Jul 16 21:50:13.667591 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:13.667395 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9fcf5399-29b4-46a1-84ed-2779ac199ead","Type":"ContainerStarted","Data":"72ddd90d0cfa02e991fb17390415fb025d04d58d7a5dc078e7ff910bb5f99889"} Jul 16 21:50:13.667591 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:13.667431 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9fcf5399-29b4-46a1-84ed-2779ac199ead","Type":"ContainerStarted","Data":"058f23bc78213220df997024a401ce63cea4e051cf6386510cb2b38df1429306"} Jul 16 21:50:13.735460 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:13.735393 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.718867305 podStartE2EDuration="2.735373573s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.506268ms" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:13.733018843 +0000 UTC m=+162.355461798" watchObservedRunningTime="2026-07-16 21:50:13.735373573 +0000 UTC m=+162.357816527" Jul 16 21:50:13.905096 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:13.905063 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5ff499df55-jnr4s"] Jul 16 21:50:13.907650 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:13.907627 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5ff499df55-jnr4s" Jul 16 21:50:13.910257 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:13.910208 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 16 21:50:13.910385 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:13.910301 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-lrkpj\"" Jul 16 21:50:13.911077 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:13.911060 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 16 21:50:13.911174 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:13.911160 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 16 21:50:13.911354 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:13.911340 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 16 21:50:13.911523 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:13.911502 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 16 21:50:13.924581 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:13.924522 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 16 21:50:13.991943 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:13.991916 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5ff499df55-jnr4s"] Jul 16 21:50:14.029653 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:14.029625 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cszs\" (UniqueName: \"kubernetes.io/projected/a35bdd2d-04ea-4c54-84e9-bece735a80ff-kube-api-access-2cszs\") pod \"telemeter-client-5ff499df55-jnr4s\" (UID: \"a35bdd2d-04ea-4c54-84e9-bece735a80ff\") " pod="openshift-monitoring/telemeter-client-5ff499df55-jnr4s" Jul 16 21:50:14.029748 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:14.029675 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a35bdd2d-04ea-4c54-84e9-bece735a80ff-federate-client-tls\") pod \"telemeter-client-5ff499df55-jnr4s\" (UID: \"a35bdd2d-04ea-4c54-84e9-bece735a80ff\") " pod="openshift-monitoring/telemeter-client-5ff499df55-jnr4s" Jul 16 21:50:14.029748 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:14.029714 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a35bdd2d-04ea-4c54-84e9-bece735a80ff-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5ff499df55-jnr4s\" (UID: \"a35bdd2d-04ea-4c54-84e9-bece735a80ff\") " pod="openshift-monitoring/telemeter-client-5ff499df55-jnr4s" Jul 16 21:50:14.029843 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:14.029763 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a35bdd2d-04ea-4c54-84e9-bece735a80ff-serving-certs-ca-bundle\") pod \"telemeter-client-5ff499df55-jnr4s\" (UID: \"a35bdd2d-04ea-4c54-84e9-bece735a80ff\") " pod="openshift-monitoring/telemeter-client-5ff499df55-jnr4s" Jul 16 21:50:14.029843 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:14.029825 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a35bdd2d-04ea-4c54-84e9-bece735a80ff-secret-telemeter-client\") pod \"telemeter-client-5ff499df55-jnr4s\" (UID: \"a35bdd2d-04ea-4c54-84e9-bece735a80ff\") " pod="openshift-monitoring/telemeter-client-5ff499df55-jnr4s" Jul 16 21:50:14.029907 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:14.029857 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a35bdd2d-04ea-4c54-84e9-bece735a80ff-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5ff499df55-jnr4s\" (UID: \"a35bdd2d-04ea-4c54-84e9-bece735a80ff\") " pod="openshift-monitoring/telemeter-client-5ff499df55-jnr4s" Jul 16 21:50:14.029907 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:14.029887 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a35bdd2d-04ea-4c54-84e9-bece735a80ff-metrics-client-ca\") pod \"telemeter-client-5ff499df55-jnr4s\" (UID: \"a35bdd2d-04ea-4c54-84e9-bece735a80ff\") " pod="openshift-monitoring/telemeter-client-5ff499df55-jnr4s" Jul 16 21:50:14.029975 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:14.029931 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a35bdd2d-04ea-4c54-84e9-bece735a80ff-telemeter-client-tls\") pod \"telemeter-client-5ff499df55-jnr4s\" (UID: \"a35bdd2d-04ea-4c54-84e9-bece735a80ff\") " pod="openshift-monitoring/telemeter-client-5ff499df55-jnr4s" Jul 16 21:50:14.131151 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:14.131126 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cszs\" (UniqueName: \"kubernetes.io/projected/a35bdd2d-04ea-4c54-84e9-bece735a80ff-kube-api-access-2cszs\") pod \"telemeter-client-5ff499df55-jnr4s\" (UID: \"a35bdd2d-04ea-4c54-84e9-bece735a80ff\") " pod="openshift-monitoring/telemeter-client-5ff499df55-jnr4s" Jul 16 21:50:14.131360 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:14.131158 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a35bdd2d-04ea-4c54-84e9-bece735a80ff-federate-client-tls\") pod \"telemeter-client-5ff499df55-jnr4s\" (UID: \"a35bdd2d-04ea-4c54-84e9-bece735a80ff\") " pod="openshift-monitoring/telemeter-client-5ff499df55-jnr4s" Jul 16 21:50:14.131360 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:14.131181 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a35bdd2d-04ea-4c54-84e9-bece735a80ff-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5ff499df55-jnr4s\" (UID: \"a35bdd2d-04ea-4c54-84e9-bece735a80ff\") " pod="openshift-monitoring/telemeter-client-5ff499df55-jnr4s" Jul 16 21:50:14.131360 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:14.131234 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a35bdd2d-04ea-4c54-84e9-bece735a80ff-serving-certs-ca-bundle\") pod \"telemeter-client-5ff499df55-jnr4s\" (UID: \"a35bdd2d-04ea-4c54-84e9-bece735a80ff\") " pod="openshift-monitoring/telemeter-client-5ff499df55-jnr4s" Jul 16 21:50:14.131360 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:14.131263 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a35bdd2d-04ea-4c54-84e9-bece735a80ff-secret-telemeter-client\") pod \"telemeter-client-5ff499df55-jnr4s\" (UID: \"a35bdd2d-04ea-4c54-84e9-bece735a80ff\") " pod="openshift-monitoring/telemeter-client-5ff499df55-jnr4s" Jul 16 21:50:14.131360 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:14.131278 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a35bdd2d-04ea-4c54-84e9-bece735a80ff-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5ff499df55-jnr4s\" (UID: \"a35bdd2d-04ea-4c54-84e9-bece735a80ff\") " pod="openshift-monitoring/telemeter-client-5ff499df55-jnr4s" Jul 16 21:50:14.131360 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:14.131296 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a35bdd2d-04ea-4c54-84e9-bece735a80ff-metrics-client-ca\") pod \"telemeter-client-5ff499df55-jnr4s\" (UID: \"a35bdd2d-04ea-4c54-84e9-bece735a80ff\") " pod="openshift-monitoring/telemeter-client-5ff499df55-jnr4s" Jul 16 21:50:14.131660 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:14.131357 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a35bdd2d-04ea-4c54-84e9-bece735a80ff-telemeter-client-tls\") pod \"telemeter-client-5ff499df55-jnr4s\" (UID: \"a35bdd2d-04ea-4c54-84e9-bece735a80ff\") " pod="openshift-monitoring/telemeter-client-5ff499df55-jnr4s" Jul 16 21:50:14.132059 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:14.132029 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a35bdd2d-04ea-4c54-84e9-bece735a80ff-serving-certs-ca-bundle\") pod \"telemeter-client-5ff499df55-jnr4s\" (UID: \"a35bdd2d-04ea-4c54-84e9-bece735a80ff\") " pod="openshift-monitoring/telemeter-client-5ff499df55-jnr4s" Jul 16 21:50:14.132153 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:14.132072 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a35bdd2d-04ea-4c54-84e9-bece735a80ff-metrics-client-ca\") pod \"telemeter-client-5ff499df55-jnr4s\" (UID: \"a35bdd2d-04ea-4c54-84e9-bece735a80ff\") " pod="openshift-monitoring/telemeter-client-5ff499df55-jnr4s" Jul 16 21:50:14.132240 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:14.132162 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a35bdd2d-04ea-4c54-84e9-bece735a80ff-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5ff499df55-jnr4s\" (UID: \"a35bdd2d-04ea-4c54-84e9-bece735a80ff\") " pod="openshift-monitoring/telemeter-client-5ff499df55-jnr4s" Jul 16 21:50:14.133639 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:14.133618 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a35bdd2d-04ea-4c54-84e9-bece735a80ff-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5ff499df55-jnr4s\" (UID: \"a35bdd2d-04ea-4c54-84e9-bece735a80ff\") " pod="openshift-monitoring/telemeter-client-5ff499df55-jnr4s" Jul 16 21:50:14.133905 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:14.133876 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a35bdd2d-04ea-4c54-84e9-bece735a80ff-secret-telemeter-client\") pod \"telemeter-client-5ff499df55-jnr4s\" (UID: \"a35bdd2d-04ea-4c54-84e9-bece735a80ff\") " pod="openshift-monitoring/telemeter-client-5ff499df55-jnr4s" Jul 16 21:50:14.133975 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:14.133945 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a35bdd2d-04ea-4c54-84e9-bece735a80ff-federate-client-tls\") pod \"telemeter-client-5ff499df55-jnr4s\" (UID: \"a35bdd2d-04ea-4c54-84e9-bece735a80ff\") " pod="openshift-monitoring/telemeter-client-5ff499df55-jnr4s" Jul 16 21:50:14.134121 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:14.134105 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a35bdd2d-04ea-4c54-84e9-bece735a80ff-telemeter-client-tls\") pod \"telemeter-client-5ff499df55-jnr4s\" (UID: \"a35bdd2d-04ea-4c54-84e9-bece735a80ff\") " pod="openshift-monitoring/telemeter-client-5ff499df55-jnr4s" Jul 16 21:50:14.143687 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:14.143663 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cszs\" (UniqueName: \"kubernetes.io/projected/a35bdd2d-04ea-4c54-84e9-bece735a80ff-kube-api-access-2cszs\") pod \"telemeter-client-5ff499df55-jnr4s\" (UID: \"a35bdd2d-04ea-4c54-84e9-bece735a80ff\") " pod="openshift-monitoring/telemeter-client-5ff499df55-jnr4s" Jul 16 21:50:14.217929 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:14.217868 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5ff499df55-jnr4s" Jul 16 21:50:14.356255 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:14.356201 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5ff499df55-jnr4s"] Jul 16 21:50:14.359073 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:50:14.359049 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda35bdd2d_04ea_4c54_84e9_bece735a80ff.slice/crio-e949a7439312418c844a6f9edfa6b9d559beb1d0657f2b6b741b435e67e03828 WatchSource:0}: Error finding container e949a7439312418c844a6f9edfa6b9d559beb1d0657f2b6b741b435e67e03828: Status 404 returned error can't find the container with id e949a7439312418c844a6f9edfa6b9d559beb1d0657f2b6b741b435e67e03828 Jul 16 21:50:14.672081 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:14.672049 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5ff499df55-jnr4s" event={"ID":"a35bdd2d-04ea-4c54-84e9-bece735a80ff","Type":"ContainerStarted","Data":"e949a7439312418c844a6f9edfa6b9d559beb1d0657f2b6b741b435e67e03828"} Jul 16 21:50:17.689817 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:17.689783 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5ff499df55-jnr4s" event={"ID":"a35bdd2d-04ea-4c54-84e9-bece735a80ff","Type":"ContainerStarted","Data":"fa931c7d7552ef71e6d7e2c39095239cc65f0f6776629f1a709556bd54ada2c5"} Jul 16 21:50:17.690323 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:17.689843 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5ff499df55-jnr4s" event={"ID":"a35bdd2d-04ea-4c54-84e9-bece735a80ff","Type":"ContainerStarted","Data":"d311587a6f3bc50d896453a33202d949e2a5eb6281f527ebcb9e7c642b2b4654"} Jul 16 21:50:17.690323 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:17.689873 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5ff499df55-jnr4s" event={"ID":"a35bdd2d-04ea-4c54-84e9-bece735a80ff","Type":"ContainerStarted","Data":"51232bc1f2f549852646f648add5754540f04baf02ce92a860835445401ae646"} Jul 16 21:50:17.715616 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:17.715557 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5ff499df55-jnr4s" podStartSLOduration=2.244094878 podStartE2EDuration="4.715539753s" totalImagesPullingTime="2.471444875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:17.713945236 +0000 UTC m=+166.336388190" watchObservedRunningTime="2026-07-16 21:50:17.715539753 +0000 UTC m=+166.337982705" Jul 16 21:50:18.594928 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:18.594882 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:50:18.699124 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:18.699094 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:50:18.774581 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:18.774552 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:50:19.319206 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:19.319169 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:50:19.388192 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:19.388137 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:50:28.932326 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:28.932250 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bbc77df58-9mpjq"] Jul 16 21:50:29.132239 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:29.132183 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:50:29.233733 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:29.233663 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:50:53.952186 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:53.952122 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-bbc77df58-9mpjq" podUID="74324078-6d2f-4256-bd15-150bf88c8f0d" containerName="console" containerID="cri-o://418b0fa854b01a31e37c566d5556d7c92893b1ede221928982a702925a5e905d" gracePeriod=15 Jul 16 21:50:54.182740 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:54.182716 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bbc77df58-9mpjq" Jul 16 21:50:54.242834 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:54.242763 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/74324078-6d2f-4256-bd15-150bf88c8f0d-oauth-serving-cert\") pod \"74324078-6d2f-4256-bd15-150bf88c8f0d\" (UID: \"74324078-6d2f-4256-bd15-150bf88c8f0d\") " Jul 16 21:50:54.242834 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:54.242803 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/74324078-6d2f-4256-bd15-150bf88c8f0d-console-serving-cert\") pod \"74324078-6d2f-4256-bd15-150bf88c8f0d\" (UID: \"74324078-6d2f-4256-bd15-150bf88c8f0d\") " Jul 16 21:50:54.243034 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:54.242994 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/74324078-6d2f-4256-bd15-150bf88c8f0d-console-config\") pod \"74324078-6d2f-4256-bd15-150bf88c8f0d\" (UID: \"74324078-6d2f-4256-bd15-150bf88c8f0d\") " Jul 16 21:50:54.243034 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:54.243030 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/74324078-6d2f-4256-bd15-150bf88c8f0d-console-oauth-config\") pod \"74324078-6d2f-4256-bd15-150bf88c8f0d\" (UID: \"74324078-6d2f-4256-bd15-150bf88c8f0d\") " Jul 16 21:50:54.243143 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:54.243060 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tsfr2\" (UniqueName: \"kubernetes.io/projected/74324078-6d2f-4256-bd15-150bf88c8f0d-kube-api-access-tsfr2\") pod \"74324078-6d2f-4256-bd15-150bf88c8f0d\" (UID: \"74324078-6d2f-4256-bd15-150bf88c8f0d\") " Jul 16 21:50:54.243143 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:54.243092 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74324078-6d2f-4256-bd15-150bf88c8f0d-trusted-ca-bundle\") pod \"74324078-6d2f-4256-bd15-150bf88c8f0d\" (UID: \"74324078-6d2f-4256-bd15-150bf88c8f0d\") " Jul 16 21:50:54.243143 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:54.243121 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/74324078-6d2f-4256-bd15-150bf88c8f0d-service-ca\") pod \"74324078-6d2f-4256-bd15-150bf88c8f0d\" (UID: \"74324078-6d2f-4256-bd15-150bf88c8f0d\") " Jul 16 21:50:54.243318 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:54.243200 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74324078-6d2f-4256-bd15-150bf88c8f0d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "74324078-6d2f-4256-bd15-150bf88c8f0d" (UID: "74324078-6d2f-4256-bd15-150bf88c8f0d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:50:54.243368 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:54.243341 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74324078-6d2f-4256-bd15-150bf88c8f0d-console-config" (OuterVolumeSpecName: "console-config") pod "74324078-6d2f-4256-bd15-150bf88c8f0d" (UID: "74324078-6d2f-4256-bd15-150bf88c8f0d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:50:54.243472 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:54.243446 2585 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/74324078-6d2f-4256-bd15-150bf88c8f0d-oauth-serving-cert\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:50:54.243539 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:54.243469 2585 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/74324078-6d2f-4256-bd15-150bf88c8f0d-console-config\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:50:54.243615 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:54.243589 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74324078-6d2f-4256-bd15-150bf88c8f0d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "74324078-6d2f-4256-bd15-150bf88c8f0d" (UID: "74324078-6d2f-4256-bd15-150bf88c8f0d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:50:54.243676 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:54.243626 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74324078-6d2f-4256-bd15-150bf88c8f0d-service-ca" (OuterVolumeSpecName: "service-ca") pod "74324078-6d2f-4256-bd15-150bf88c8f0d" (UID: "74324078-6d2f-4256-bd15-150bf88c8f0d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:50:54.245056 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:54.245036 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74324078-6d2f-4256-bd15-150bf88c8f0d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "74324078-6d2f-4256-bd15-150bf88c8f0d" (UID: "74324078-6d2f-4256-bd15-150bf88c8f0d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:50:54.245433 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:54.245403 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74324078-6d2f-4256-bd15-150bf88c8f0d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "74324078-6d2f-4256-bd15-150bf88c8f0d" (UID: "74324078-6d2f-4256-bd15-150bf88c8f0d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:50:54.245433 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:54.245427 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74324078-6d2f-4256-bd15-150bf88c8f0d-kube-api-access-tsfr2" (OuterVolumeSpecName: "kube-api-access-tsfr2") pod "74324078-6d2f-4256-bd15-150bf88c8f0d" (UID: "74324078-6d2f-4256-bd15-150bf88c8f0d"). InnerVolumeSpecName "kube-api-access-tsfr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:50:54.344428 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:54.344397 2585 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/74324078-6d2f-4256-bd15-150bf88c8f0d-console-oauth-config\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:50:54.344428 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:54.344423 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsfr2\" (UniqueName: \"kubernetes.io/projected/74324078-6d2f-4256-bd15-150bf88c8f0d-kube-api-access-tsfr2\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:50:54.344610 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:54.344435 2585 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74324078-6d2f-4256-bd15-150bf88c8f0d-trusted-ca-bundle\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:50:54.344610 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:54.344447 2585 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/74324078-6d2f-4256-bd15-150bf88c8f0d-service-ca\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:50:54.344610 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:54.344459 2585 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/74324078-6d2f-4256-bd15-150bf88c8f0d-console-serving-cert\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:50:54.395865 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:54.395822 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:50:54.827292 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:54.827256 2585 generic.go:356] "Generic (PLEG): container finished" podID="74324078-6d2f-4256-bd15-150bf88c8f0d" containerID="418b0fa854b01a31e37c566d5556d7c92893b1ede221928982a702925a5e905d" exitCode=0 Jul 16 21:50:54.827457 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:54.827300 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bbc77df58-9mpjq" Jul 16 21:50:54.827457 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:54.827323 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bbc77df58-9mpjq" event={"ID":"74324078-6d2f-4256-bd15-150bf88c8f0d","Type":"ContainerDied","Data":"418b0fa854b01a31e37c566d5556d7c92893b1ede221928982a702925a5e905d"} Jul 16 21:50:54.827457 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:54.827367 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bbc77df58-9mpjq" event={"ID":"74324078-6d2f-4256-bd15-150bf88c8f0d","Type":"ContainerDied","Data":"9ab854505f848cadf125e67b2f6671f47b4234dc71846b09799624f8cca685d2"} Jul 16 21:50:54.827457 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:54.827389 2585 scope.go:117] "RemoveContainer" containerID="418b0fa854b01a31e37c566d5556d7c92893b1ede221928982a702925a5e905d" Jul 16 21:50:54.836460 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:54.836439 2585 scope.go:117] "RemoveContainer" containerID="418b0fa854b01a31e37c566d5556d7c92893b1ede221928982a702925a5e905d" Jul 16 21:50:54.836761 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:50:54.836740 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"418b0fa854b01a31e37c566d5556d7c92893b1ede221928982a702925a5e905d\": container with ID starting with 418b0fa854b01a31e37c566d5556d7c92893b1ede221928982a702925a5e905d not found: ID does not exist" containerID="418b0fa854b01a31e37c566d5556d7c92893b1ede221928982a702925a5e905d" Jul 16 21:50:54.836810 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:54.836780 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"418b0fa854b01a31e37c566d5556d7c92893b1ede221928982a702925a5e905d"} err="failed to get container status \"418b0fa854b01a31e37c566d5556d7c92893b1ede221928982a702925a5e905d\": rpc error: code = NotFound desc = could not find container \"418b0fa854b01a31e37c566d5556d7c92893b1ede221928982a702925a5e905d\": container with ID starting with 418b0fa854b01a31e37c566d5556d7c92893b1ede221928982a702925a5e905d not found: ID does not exist" Jul 16 21:50:54.849884 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:54.849858 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bbc77df58-9mpjq"] Jul 16 21:50:54.853640 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:54.853622 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-bbc77df58-9mpjq"] Jul 16 21:50:55.932880 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:50:55.932851 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74324078-6d2f-4256-bd15-150bf88c8f0d" path="/var/lib/kubelet/pods/74324078-6d2f-4256-bd15-150bf88c8f0d/volumes" Jul 16 21:51:04.555088 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:51:04.555048 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:51:24.139024 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:51:24.138986 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:51:24.327944 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:51:24.327914 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:51:30.913389 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:51:30.913350 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:51:31.257044 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:51:31.256963 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:51:36.088059 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:51:36.088027 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:51:36.222112 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:51:36.222069 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:51:42.352129 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:51:42.352093 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:51:42.453685 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:51:42.453659 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:51:42.553339 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:51:42.553298 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:51:48.178381 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:51:48.178348 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:51:48.281037 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:51:48.281000 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:52:29.323455 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:29.323421 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:52:30.349594 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:30.349554 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d772v"] Jul 16 21:52:30.350053 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:30.350001 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74324078-6d2f-4256-bd15-150bf88c8f0d" containerName="console" Jul 16 21:52:30.350053 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:30.350018 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="74324078-6d2f-4256-bd15-150bf88c8f0d" containerName="console" Jul 16 21:52:30.350170 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:30.350106 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="74324078-6d2f-4256-bd15-150bf88c8f0d" containerName="console" Jul 16 21:52:30.353092 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:30.353070 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d772v" Jul 16 21:52:30.355900 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:30.355880 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 16 21:52:30.356021 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:30.355929 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 16 21:52:30.356021 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:30.355993 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-qsqxh\"" Jul 16 21:52:30.366868 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:30.366849 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d772v"] Jul 16 21:52:30.390422 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:30.390392 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf7rf\" (UniqueName: \"kubernetes.io/projected/264a2e2f-a7db-4a3f-808d-b8200743e4b2-kube-api-access-kf7rf\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d772v\" (UID: \"264a2e2f-a7db-4a3f-808d-b8200743e4b2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d772v" Jul 16 21:52:30.390422 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:30.390423 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/264a2e2f-a7db-4a3f-808d-b8200743e4b2-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d772v\" (UID: \"264a2e2f-a7db-4a3f-808d-b8200743e4b2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d772v" Jul 16 21:52:30.390573 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:30.390458 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/264a2e2f-a7db-4a3f-808d-b8200743e4b2-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d772v\" (UID: \"264a2e2f-a7db-4a3f-808d-b8200743e4b2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d772v" Jul 16 21:52:30.491115 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:30.491086 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf7rf\" (UniqueName: \"kubernetes.io/projected/264a2e2f-a7db-4a3f-808d-b8200743e4b2-kube-api-access-kf7rf\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d772v\" (UID: \"264a2e2f-a7db-4a3f-808d-b8200743e4b2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d772v" Jul 16 21:52:30.491115 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:30.491119 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/264a2e2f-a7db-4a3f-808d-b8200743e4b2-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d772v\" (UID: \"264a2e2f-a7db-4a3f-808d-b8200743e4b2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d772v" Jul 16 21:52:30.491312 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:30.491155 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/264a2e2f-a7db-4a3f-808d-b8200743e4b2-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d772v\" (UID: \"264a2e2f-a7db-4a3f-808d-b8200743e4b2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d772v" Jul 16 21:52:30.491486 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:30.491467 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/264a2e2f-a7db-4a3f-808d-b8200743e4b2-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d772v\" (UID: \"264a2e2f-a7db-4a3f-808d-b8200743e4b2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d772v" Jul 16 21:52:30.491526 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:30.491493 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/264a2e2f-a7db-4a3f-808d-b8200743e4b2-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d772v\" (UID: \"264a2e2f-a7db-4a3f-808d-b8200743e4b2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d772v" Jul 16 21:52:30.500570 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:30.500545 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf7rf\" (UniqueName: \"kubernetes.io/projected/264a2e2f-a7db-4a3f-808d-b8200743e4b2-kube-api-access-kf7rf\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d772v\" (UID: \"264a2e2f-a7db-4a3f-808d-b8200743e4b2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d772v" Jul 16 21:52:30.616744 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:30.616621 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:52:30.661585 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:30.661554 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d772v" Jul 16 21:52:30.789613 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:30.789591 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d772v"] Jul 16 21:52:30.792344 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:52:30.792316 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod264a2e2f_a7db_4a3f_808d_b8200743e4b2.slice/crio-a14bafc002dc25c623b6c712540dacd5c5e6f873cfc0a61023fa7670663f8fc4 WatchSource:0}: Error finding container a14bafc002dc25c623b6c712540dacd5c5e6f873cfc0a61023fa7670663f8fc4: Status 404 returned error can't find the container with id a14bafc002dc25c623b6c712540dacd5c5e6f873cfc0a61023fa7670663f8fc4 Jul 16 21:52:31.175028 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:31.174993 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d772v" event={"ID":"264a2e2f-a7db-4a3f-808d-b8200743e4b2","Type":"ContainerStarted","Data":"a14bafc002dc25c623b6c712540dacd5c5e6f873cfc0a61023fa7670663f8fc4"} Jul 16 21:52:31.757673 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:31.757647 2585 kubelet.go:1648] "Image garbage collection succeeded" Jul 16 21:52:37.200883 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:37.200848 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d772v" event={"ID":"264a2e2f-a7db-4a3f-808d-b8200743e4b2","Type":"ContainerStarted","Data":"d59f658bf499689ae91306a240af49938defd73d7937a6f9846c18acc9a23ca2"} Jul 16 21:52:38.205500 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:38.205452 2585 generic.go:356] "Generic (PLEG): container finished" podID="264a2e2f-a7db-4a3f-808d-b8200743e4b2" containerID="d59f658bf499689ae91306a240af49938defd73d7937a6f9846c18acc9a23ca2" exitCode=0 Jul 16 21:52:38.205879 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:38.205542 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d772v" event={"ID":"264a2e2f-a7db-4a3f-808d-b8200743e4b2","Type":"ContainerDied","Data":"d59f658bf499689ae91306a240af49938defd73d7937a6f9846c18acc9a23ca2"} Jul 16 21:52:38.206595 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:38.206580 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 21:52:41.217730 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:41.217690 2585 generic.go:356] "Generic (PLEG): container finished" podID="264a2e2f-a7db-4a3f-808d-b8200743e4b2" containerID="96d4f8d0ae4b0ea94d451e4e05b016e9f00cb0a7e34975222fa7fdd3cd297ba9" exitCode=0 Jul 16 21:52:41.218129 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:41.217774 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d772v" event={"ID":"264a2e2f-a7db-4a3f-808d-b8200743e4b2","Type":"ContainerDied","Data":"96d4f8d0ae4b0ea94d451e4e05b016e9f00cb0a7e34975222fa7fdd3cd297ba9"} Jul 16 21:52:48.249119 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:48.249092 2585 generic.go:356] "Generic (PLEG): container finished" podID="264a2e2f-a7db-4a3f-808d-b8200743e4b2" containerID="66aeec6f5e7560ff10d5bf024858474d72530e2eb6d334497cc1b7c256af4546" exitCode=0 Jul 16 21:52:48.249438 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:48.249132 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d772v" event={"ID":"264a2e2f-a7db-4a3f-808d-b8200743e4b2","Type":"ContainerDied","Data":"66aeec6f5e7560ff10d5bf024858474d72530e2eb6d334497cc1b7c256af4546"} Jul 16 21:52:49.378619 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:49.378594 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d772v" Jul 16 21:52:49.458274 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:49.458247 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kf7rf\" (UniqueName: \"kubernetes.io/projected/264a2e2f-a7db-4a3f-808d-b8200743e4b2-kube-api-access-kf7rf\") pod \"264a2e2f-a7db-4a3f-808d-b8200743e4b2\" (UID: \"264a2e2f-a7db-4a3f-808d-b8200743e4b2\") " Jul 16 21:52:49.458434 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:49.458323 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/264a2e2f-a7db-4a3f-808d-b8200743e4b2-bundle\") pod \"264a2e2f-a7db-4a3f-808d-b8200743e4b2\" (UID: \"264a2e2f-a7db-4a3f-808d-b8200743e4b2\") " Jul 16 21:52:49.458434 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:49.458344 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/264a2e2f-a7db-4a3f-808d-b8200743e4b2-util\") pod \"264a2e2f-a7db-4a3f-808d-b8200743e4b2\" (UID: \"264a2e2f-a7db-4a3f-808d-b8200743e4b2\") " Jul 16 21:52:49.459011 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:49.458983 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/264a2e2f-a7db-4a3f-808d-b8200743e4b2-bundle" (OuterVolumeSpecName: "bundle") pod "264a2e2f-a7db-4a3f-808d-b8200743e4b2" (UID: "264a2e2f-a7db-4a3f-808d-b8200743e4b2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:52:49.460467 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:49.460440 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/264a2e2f-a7db-4a3f-808d-b8200743e4b2-kube-api-access-kf7rf" (OuterVolumeSpecName: "kube-api-access-kf7rf") pod "264a2e2f-a7db-4a3f-808d-b8200743e4b2" (UID: "264a2e2f-a7db-4a3f-808d-b8200743e4b2"). InnerVolumeSpecName "kube-api-access-kf7rf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:52:49.463940 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:49.463915 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/264a2e2f-a7db-4a3f-808d-b8200743e4b2-util" (OuterVolumeSpecName: "util") pod "264a2e2f-a7db-4a3f-808d-b8200743e4b2" (UID: "264a2e2f-a7db-4a3f-808d-b8200743e4b2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:52:49.559494 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:49.559473 2585 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/264a2e2f-a7db-4a3f-808d-b8200743e4b2-bundle\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:52:49.559494 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:49.559493 2585 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/264a2e2f-a7db-4a3f-808d-b8200743e4b2-util\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:52:49.559614 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:49.559502 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf7rf\" (UniqueName: \"kubernetes.io/projected/264a2e2f-a7db-4a3f-808d-b8200743e4b2-kube-api-access-kf7rf\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:52:50.258235 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:50.258140 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d772v" Jul 16 21:52:50.258378 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:50.258138 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d772v" event={"ID":"264a2e2f-a7db-4a3f-808d-b8200743e4b2","Type":"ContainerDied","Data":"a14bafc002dc25c623b6c712540dacd5c5e6f873cfc0a61023fa7670663f8fc4"} Jul 16 21:52:50.258378 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:50.258272 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a14bafc002dc25c623b6c712540dacd5c5e6f873cfc0a61023fa7670663f8fc4" Jul 16 21:52:51.355443 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:51.355402 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:52:52.182602 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:52.182563 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:52:53.491797 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:53.491766 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-s6thm"] Jul 16 21:52:53.492194 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:53.492034 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="264a2e2f-a7db-4a3f-808d-b8200743e4b2" containerName="util" Jul 16 21:52:53.492194 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:53.492044 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="264a2e2f-a7db-4a3f-808d-b8200743e4b2" containerName="util" Jul 16 21:52:53.492194 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:53.492064 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="264a2e2f-a7db-4a3f-808d-b8200743e4b2" containerName="extract" Jul 16 21:52:53.492194 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:53.492070 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="264a2e2f-a7db-4a3f-808d-b8200743e4b2" containerName="extract" Jul 16 21:52:53.492194 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:53.492079 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="264a2e2f-a7db-4a3f-808d-b8200743e4b2" containerName="pull" Jul 16 21:52:53.492194 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:53.492084 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="264a2e2f-a7db-4a3f-808d-b8200743e4b2" containerName="pull" Jul 16 21:52:53.492194 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:53.492131 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="264a2e2f-a7db-4a3f-808d-b8200743e4b2" containerName="extract" Jul 16 21:52:53.495480 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:53.495461 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-s6thm" Jul 16 21:52:53.498319 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:53.498296 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 16 21:52:53.498429 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:53.498400 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 16 21:52:53.498839 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:53.498825 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-g679h\"" Jul 16 21:52:53.518384 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:53.518361 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-s6thm"] Jul 16 21:52:53.590974 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:53.590942 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b9e9305-6227-4e4c-8e74-5729adbe6ee3-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-s6thm\" (UID: \"1b9e9305-6227-4e4c-8e74-5729adbe6ee3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-s6thm" Jul 16 21:52:53.591077 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:53.590991 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg2wp\" (UniqueName: \"kubernetes.io/projected/1b9e9305-6227-4e4c-8e74-5729adbe6ee3-kube-api-access-gg2wp\") pod \"cert-manager-operator-controller-manager-55c444bf86-s6thm\" (UID: \"1b9e9305-6227-4e4c-8e74-5729adbe6ee3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-s6thm" Jul 16 21:52:53.692051 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:53.692020 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b9e9305-6227-4e4c-8e74-5729adbe6ee3-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-s6thm\" (UID: \"1b9e9305-6227-4e4c-8e74-5729adbe6ee3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-s6thm" Jul 16 21:52:53.692180 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:53.692072 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg2wp\" (UniqueName: \"kubernetes.io/projected/1b9e9305-6227-4e4c-8e74-5729adbe6ee3-kube-api-access-gg2wp\") pod \"cert-manager-operator-controller-manager-55c444bf86-s6thm\" (UID: \"1b9e9305-6227-4e4c-8e74-5729adbe6ee3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-s6thm" Jul 16 21:52:53.692401 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:53.692380 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b9e9305-6227-4e4c-8e74-5729adbe6ee3-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-s6thm\" (UID: \"1b9e9305-6227-4e4c-8e74-5729adbe6ee3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-s6thm" Jul 16 21:52:53.706734 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:53.706707 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg2wp\" (UniqueName: \"kubernetes.io/projected/1b9e9305-6227-4e4c-8e74-5729adbe6ee3-kube-api-access-gg2wp\") pod \"cert-manager-operator-controller-manager-55c444bf86-s6thm\" (UID: \"1b9e9305-6227-4e4c-8e74-5729adbe6ee3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-s6thm" Jul 16 21:52:53.804323 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:53.804303 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-s6thm" Jul 16 21:52:53.955680 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:53.955651 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-s6thm"] Jul 16 21:52:53.959179 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:52:53.959147 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b9e9305_6227_4e4c_8e74_5729adbe6ee3.slice/crio-73a3d9306e4b3209c807421ef13ae0b8b503feaf4607b797b4b039c4b3459326 WatchSource:0}: Error finding container 73a3d9306e4b3209c807421ef13ae0b8b503feaf4607b797b4b039c4b3459326: Status 404 returned error can't find the container with id 73a3d9306e4b3209c807421ef13ae0b8b503feaf4607b797b4b039c4b3459326 Jul 16 21:52:54.274814 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:54.274745 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-s6thm" event={"ID":"1b9e9305-6227-4e4c-8e74-5729adbe6ee3","Type":"ContainerStarted","Data":"73a3d9306e4b3209c807421ef13ae0b8b503feaf4607b797b4b039c4b3459326"} Jul 16 21:52:55.200302 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:55.200264 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:52:55.304007 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:55.303948 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:52:56.284310 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:56.284269 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-s6thm" event={"ID":"1b9e9305-6227-4e4c-8e74-5729adbe6ee3","Type":"ContainerStarted","Data":"dca2417b2384bc1bd0cda8c74985633302b7f5092ae3de08707266d66e818539"} Jul 16 21:52:56.340520 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:56.340474 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-s6thm" podStartSLOduration=1.782507223 podStartE2EDuration="3.340459451s" totalImagesPullingTime="1.557952228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:52:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:52:56.336735999 +0000 UTC m=+324.959178952" watchObservedRunningTime="2026-07-16 21:52:56.340459451 +0000 UTC m=+324.962902404" Jul 16 21:52:56.725089 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:56.725048 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:52:56.884012 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:56.883974 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:52:57.311864 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:57.311831 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:52:57.431892 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:57.431862 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fxxn"] Jul 16 21:52:57.435423 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:57.435407 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fxxn" Jul 16 21:52:57.440894 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:57.440865 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-qsqxh\"" Jul 16 21:52:57.440894 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:57.440865 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 16 21:52:57.441856 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:57.441741 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 16 21:52:57.450359 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:57.450333 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fxxn"] Jul 16 21:52:57.483612 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:57.483588 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:52:57.523605 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:57.523583 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxqk6\" (UniqueName: \"kubernetes.io/projected/46a7a543-3059-4082-a9ef-693254bb0505-kube-api-access-hxqk6\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fxxn\" (UID: \"46a7a543-3059-4082-a9ef-693254bb0505\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fxxn" Jul 16 21:52:57.523708 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:57.523621 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46a7a543-3059-4082-a9ef-693254bb0505-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fxxn\" (UID: \"46a7a543-3059-4082-a9ef-693254bb0505\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fxxn" Jul 16 21:52:57.523708 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:57.523641 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46a7a543-3059-4082-a9ef-693254bb0505-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fxxn\" (UID: \"46a7a543-3059-4082-a9ef-693254bb0505\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fxxn" Jul 16 21:52:57.624232 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:57.624157 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxqk6\" (UniqueName: \"kubernetes.io/projected/46a7a543-3059-4082-a9ef-693254bb0505-kube-api-access-hxqk6\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fxxn\" (UID: \"46a7a543-3059-4082-a9ef-693254bb0505\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fxxn" Jul 16 21:52:57.624232 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:57.624195 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46a7a543-3059-4082-a9ef-693254bb0505-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fxxn\" (UID: \"46a7a543-3059-4082-a9ef-693254bb0505\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fxxn" Jul 16 21:52:57.624420 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:57.624240 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46a7a543-3059-4082-a9ef-693254bb0505-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fxxn\" (UID: \"46a7a543-3059-4082-a9ef-693254bb0505\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fxxn" Jul 16 21:52:57.624635 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:57.624614 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46a7a543-3059-4082-a9ef-693254bb0505-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fxxn\" (UID: \"46a7a543-3059-4082-a9ef-693254bb0505\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fxxn" Jul 16 21:52:57.624635 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:57.624629 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46a7a543-3059-4082-a9ef-693254bb0505-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fxxn\" (UID: \"46a7a543-3059-4082-a9ef-693254bb0505\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fxxn" Jul 16 21:52:57.636658 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:57.636638 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxqk6\" (UniqueName: \"kubernetes.io/projected/46a7a543-3059-4082-a9ef-693254bb0505-kube-api-access-hxqk6\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fxxn\" (UID: \"46a7a543-3059-4082-a9ef-693254bb0505\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fxxn" Jul 16 21:52:57.744879 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:57.744848 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fxxn" Jul 16 21:52:57.882282 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:57.882261 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fxxn"] Jul 16 21:52:57.884152 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:52:57.884127 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46a7a543_3059_4082_a9ef_693254bb0505.slice/crio-e1299c6b9c63699b56ca55d2efa138c0f63cbb2ad32bb239e963a489003be732 WatchSource:0}: Error finding container e1299c6b9c63699b56ca55d2efa138c0f63cbb2ad32bb239e963a489003be732: Status 404 returned error can't find the container with id e1299c6b9c63699b56ca55d2efa138c0f63cbb2ad32bb239e963a489003be732 Jul 16 21:52:58.294141 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:58.294047 2585 generic.go:356] "Generic (PLEG): container finished" podID="46a7a543-3059-4082-a9ef-693254bb0505" containerID="c246fae6c5ae810c401a814bb35f55b5a64098ccbdc943cf1f52c6f641c7fe56" exitCode=0 Jul 16 21:52:58.294302 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:58.294134 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fxxn" event={"ID":"46a7a543-3059-4082-a9ef-693254bb0505","Type":"ContainerDied","Data":"c246fae6c5ae810c401a814bb35f55b5a64098ccbdc943cf1f52c6f641c7fe56"} Jul 16 21:52:58.294302 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:58.294184 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fxxn" event={"ID":"46a7a543-3059-4082-a9ef-693254bb0505","Type":"ContainerStarted","Data":"e1299c6b9c63699b56ca55d2efa138c0f63cbb2ad32bb239e963a489003be732"} Jul 16 21:52:59.786085 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:52:59.786023 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:00.023056 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:00.023001 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:00.304582 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:00.304552 2585 generic.go:356] "Generic (PLEG): container finished" podID="46a7a543-3059-4082-a9ef-693254bb0505" containerID="6205b091e21b1514337d7ba38ce3d547f9693cc8f8f2f6084341507b708a4d47" exitCode=0 Jul 16 21:53:00.304716 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:00.304620 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fxxn" event={"ID":"46a7a543-3059-4082-a9ef-693254bb0505","Type":"ContainerDied","Data":"6205b091e21b1514337d7ba38ce3d547f9693cc8f8f2f6084341507b708a4d47"} Jul 16 21:53:00.353237 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:00.353200 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-kbnb5"] Jul 16 21:53:00.357252 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:00.357231 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-kbnb5" Jul 16 21:53:00.360170 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:00.360150 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 16 21:53:00.360286 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:00.360204 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 16 21:53:00.360286 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:00.360161 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-chvll\"" Jul 16 21:53:00.370068 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:00.370044 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-kbnb5"] Jul 16 21:53:00.448361 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:00.448333 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nxhk\" (UniqueName: \"kubernetes.io/projected/6d9694ea-3f19-4c34-887d-647b40ff1761-kube-api-access-2nxhk\") pod \"cert-manager-cainjector-559cdd8c87-kbnb5\" (UID: \"6d9694ea-3f19-4c34-887d-647b40ff1761\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kbnb5" Jul 16 21:53:00.448470 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:00.448379 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d9694ea-3f19-4c34-887d-647b40ff1761-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-kbnb5\" (UID: \"6d9694ea-3f19-4c34-887d-647b40ff1761\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kbnb5" Jul 16 21:53:00.549816 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:00.549788 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nxhk\" (UniqueName: \"kubernetes.io/projected/6d9694ea-3f19-4c34-887d-647b40ff1761-kube-api-access-2nxhk\") pod \"cert-manager-cainjector-559cdd8c87-kbnb5\" (UID: \"6d9694ea-3f19-4c34-887d-647b40ff1761\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kbnb5" Jul 16 21:53:00.549920 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:00.549824 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d9694ea-3f19-4c34-887d-647b40ff1761-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-kbnb5\" (UID: \"6d9694ea-3f19-4c34-887d-647b40ff1761\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kbnb5" Jul 16 21:53:00.558590 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:00.558565 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d9694ea-3f19-4c34-887d-647b40ff1761-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-kbnb5\" (UID: \"6d9694ea-3f19-4c34-887d-647b40ff1761\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kbnb5" Jul 16 21:53:00.558714 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:00.558695 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nxhk\" (UniqueName: \"kubernetes.io/projected/6d9694ea-3f19-4c34-887d-647b40ff1761-kube-api-access-2nxhk\") pod \"cert-manager-cainjector-559cdd8c87-kbnb5\" (UID: \"6d9694ea-3f19-4c34-887d-647b40ff1761\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kbnb5" Jul 16 21:53:00.665778 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:00.665740 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-kbnb5" Jul 16 21:53:00.808182 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:00.808159 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-kbnb5"] Jul 16 21:53:00.810479 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:53:00.810453 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d9694ea_3f19_4c34_887d_647b40ff1761.slice/crio-e61302606d04de263ddd7e62ccc5a9c12ca62b88de8b7cdd20ff312a1f9c305a WatchSource:0}: Error finding container e61302606d04de263ddd7e62ccc5a9c12ca62b88de8b7cdd20ff312a1f9c305a: Status 404 returned error can't find the container with id e61302606d04de263ddd7e62ccc5a9c12ca62b88de8b7cdd20ff312a1f9c305a Jul 16 21:53:01.312248 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:01.312118 2585 generic.go:356] "Generic (PLEG): container finished" podID="46a7a543-3059-4082-a9ef-693254bb0505" containerID="5050443c45b057b98265a79d5c1f7015838a23ccabee67627f2dd6b1b58c7612" exitCode=0 Jul 16 21:53:01.312248 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:01.312172 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fxxn" event={"ID":"46a7a543-3059-4082-a9ef-693254bb0505","Type":"ContainerDied","Data":"5050443c45b057b98265a79d5c1f7015838a23ccabee67627f2dd6b1b58c7612"} Jul 16 21:53:01.313509 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:01.313484 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-kbnb5" event={"ID":"6d9694ea-3f19-4c34-887d-647b40ff1761","Type":"ContainerStarted","Data":"e61302606d04de263ddd7e62ccc5a9c12ca62b88de8b7cdd20ff312a1f9c305a"} Jul 16 21:53:02.069967 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:02.069928 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:02.176210 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:02.176176 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:03.177815 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:03.177794 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fxxn" Jul 16 21:53:03.273108 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:03.273081 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxqk6\" (UniqueName: \"kubernetes.io/projected/46a7a543-3059-4082-a9ef-693254bb0505-kube-api-access-hxqk6\") pod \"46a7a543-3059-4082-a9ef-693254bb0505\" (UID: \"46a7a543-3059-4082-a9ef-693254bb0505\") " Jul 16 21:53:03.273242 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:03.273129 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46a7a543-3059-4082-a9ef-693254bb0505-util\") pod \"46a7a543-3059-4082-a9ef-693254bb0505\" (UID: \"46a7a543-3059-4082-a9ef-693254bb0505\") " Jul 16 21:53:03.273309 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:03.273268 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46a7a543-3059-4082-a9ef-693254bb0505-bundle\") pod \"46a7a543-3059-4082-a9ef-693254bb0505\" (UID: \"46a7a543-3059-4082-a9ef-693254bb0505\") " Jul 16 21:53:03.273655 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:03.273615 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46a7a543-3059-4082-a9ef-693254bb0505-bundle" (OuterVolumeSpecName: "bundle") pod "46a7a543-3059-4082-a9ef-693254bb0505" (UID: "46a7a543-3059-4082-a9ef-693254bb0505"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:53:03.275259 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:03.275211 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46a7a543-3059-4082-a9ef-693254bb0505-kube-api-access-hxqk6" (OuterVolumeSpecName: "kube-api-access-hxqk6") pod "46a7a543-3059-4082-a9ef-693254bb0505" (UID: "46a7a543-3059-4082-a9ef-693254bb0505"). InnerVolumeSpecName "kube-api-access-hxqk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:53:03.279441 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:03.279418 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46a7a543-3059-4082-a9ef-693254bb0505-util" (OuterVolumeSpecName: "util") pod "46a7a543-3059-4082-a9ef-693254bb0505" (UID: "46a7a543-3059-4082-a9ef-693254bb0505"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:53:03.325056 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:03.325025 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-kbnb5" event={"ID":"6d9694ea-3f19-4c34-887d-647b40ff1761","Type":"ContainerStarted","Data":"8e5737917c6762acbd17f412889b0ef12971edf64bdb72ed4bf63bed6f17a2ae"} Jul 16 21:53:03.326982 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:03.326960 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fxxn" event={"ID":"46a7a543-3059-4082-a9ef-693254bb0505","Type":"ContainerDied","Data":"e1299c6b9c63699b56ca55d2efa138c0f63cbb2ad32bb239e963a489003be732"} Jul 16 21:53:03.327096 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:03.326998 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1299c6b9c63699b56ca55d2efa138c0f63cbb2ad32bb239e963a489003be732" Jul 16 21:53:03.327096 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:03.327000 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2fxxn" Jul 16 21:53:03.342658 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:03.342616 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-kbnb5" podStartSLOduration=0.920244789 podStartE2EDuration="3.34260198s" totalImagesPullingTime="2.422357191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:53:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:53:03.340608095 +0000 UTC m=+331.963051050" watchObservedRunningTime="2026-07-16 21:53:03.34260198 +0000 UTC m=+331.965044935" Jul 16 21:53:03.374621 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:03.374588 2585 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46a7a543-3059-4082-a9ef-693254bb0505-bundle\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:53:03.374621 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:03.374615 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxqk6\" (UniqueName: \"kubernetes.io/projected/46a7a543-3059-4082-a9ef-693254bb0505-kube-api-access-hxqk6\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:53:03.374621 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:03.374625 2585 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46a7a543-3059-4082-a9ef-693254bb0505-util\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:53:07.216292 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:07.216255 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:07.514210 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:07.514134 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-2prkf"] Jul 16 21:53:07.514480 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:07.514466 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46a7a543-3059-4082-a9ef-693254bb0505" containerName="util" Jul 16 21:53:07.514524 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:07.514482 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="46a7a543-3059-4082-a9ef-693254bb0505" containerName="util" Jul 16 21:53:07.514524 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:07.514498 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46a7a543-3059-4082-a9ef-693254bb0505" containerName="pull" Jul 16 21:53:07.514524 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:07.514504 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="46a7a543-3059-4082-a9ef-693254bb0505" containerName="pull" Jul 16 21:53:07.514524 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:07.514513 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46a7a543-3059-4082-a9ef-693254bb0505" containerName="extract" Jul 16 21:53:07.514524 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:07.514519 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="46a7a543-3059-4082-a9ef-693254bb0505" containerName="extract" Jul 16 21:53:07.514671 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:07.514567 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="46a7a543-3059-4082-a9ef-693254bb0505" containerName="extract" Jul 16 21:53:07.517498 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:07.517482 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-2prkf" Jul 16 21:53:07.519878 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:07.519855 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-lrgb5\"" Jul 16 21:53:07.528266 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:07.528244 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-2prkf"] Jul 16 21:53:07.609613 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:07.609591 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/636d8f0a-56e6-425b-94f9-38eee5662cdd-bound-sa-token\") pod \"cert-manager-55b4d954d5-2prkf\" (UID: \"636d8f0a-56e6-425b-94f9-38eee5662cdd\") " pod="cert-manager/cert-manager-55b4d954d5-2prkf" Jul 16 21:53:07.609720 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:07.609626 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqwkg\" (UniqueName: \"kubernetes.io/projected/636d8f0a-56e6-425b-94f9-38eee5662cdd-kube-api-access-vqwkg\") pod \"cert-manager-55b4d954d5-2prkf\" (UID: \"636d8f0a-56e6-425b-94f9-38eee5662cdd\") " pod="cert-manager/cert-manager-55b4d954d5-2prkf" Jul 16 21:53:07.710964 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:07.710940 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/636d8f0a-56e6-425b-94f9-38eee5662cdd-bound-sa-token\") pod \"cert-manager-55b4d954d5-2prkf\" (UID: \"636d8f0a-56e6-425b-94f9-38eee5662cdd\") " pod="cert-manager/cert-manager-55b4d954d5-2prkf" Jul 16 21:53:07.711058 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:07.710976 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqwkg\" (UniqueName: \"kubernetes.io/projected/636d8f0a-56e6-425b-94f9-38eee5662cdd-kube-api-access-vqwkg\") pod \"cert-manager-55b4d954d5-2prkf\" (UID: \"636d8f0a-56e6-425b-94f9-38eee5662cdd\") " pod="cert-manager/cert-manager-55b4d954d5-2prkf" Jul 16 21:53:07.719502 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:07.719475 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/636d8f0a-56e6-425b-94f9-38eee5662cdd-bound-sa-token\") pod \"cert-manager-55b4d954d5-2prkf\" (UID: \"636d8f0a-56e6-425b-94f9-38eee5662cdd\") " pod="cert-manager/cert-manager-55b4d954d5-2prkf" Jul 16 21:53:07.719660 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:07.719643 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqwkg\" (UniqueName: \"kubernetes.io/projected/636d8f0a-56e6-425b-94f9-38eee5662cdd-kube-api-access-vqwkg\") pod \"cert-manager-55b4d954d5-2prkf\" (UID: \"636d8f0a-56e6-425b-94f9-38eee5662cdd\") " pod="cert-manager/cert-manager-55b4d954d5-2prkf" Jul 16 21:53:07.826296 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:07.826269 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-2prkf" Jul 16 21:53:07.971170 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:07.971141 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-2prkf"] Jul 16 21:53:07.973514 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:53:07.973488 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod636d8f0a_56e6_425b_94f9_38eee5662cdd.slice/crio-885fc447238e23b3e2ac6321ec1eec7951cd64820ff58e3539102d357d48865d WatchSource:0}: Error finding container 885fc447238e23b3e2ac6321ec1eec7951cd64820ff58e3539102d357d48865d: Status 404 returned error can't find the container with id 885fc447238e23b3e2ac6321ec1eec7951cd64820ff58e3539102d357d48865d Jul 16 21:53:08.349863 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:08.349826 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-2prkf" event={"ID":"636d8f0a-56e6-425b-94f9-38eee5662cdd","Type":"ContainerStarted","Data":"8306bfa1db4ba69bbd89056cfbd291970c6ff0dae19a5abae6db6f443159165c"} Jul 16 21:53:08.349863 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:08.349866 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-2prkf" event={"ID":"636d8f0a-56e6-425b-94f9-38eee5662cdd","Type":"ContainerStarted","Data":"885fc447238e23b3e2ac6321ec1eec7951cd64820ff58e3539102d357d48865d"} Jul 16 21:53:08.367455 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:08.367410 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-2prkf" podStartSLOduration=1.36737354 podStartE2EDuration="1.36737354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:53:08.366490287 +0000 UTC m=+336.988933237" watchObservedRunningTime="2026-07-16 21:53:08.36737354 +0000 UTC m=+336.989816493" Jul 16 21:53:09.050921 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:09.050879 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:09.235854 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:09.235827 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:15.289057 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:15.289024 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:15.387737 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:15.387701 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:16.923146 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:16.923118 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdqfc5"] Jul 16 21:53:16.930934 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:16.930916 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdqfc5" Jul 16 21:53:16.933596 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:16.933561 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 16 21:53:16.933726 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:16.933654 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-qsqxh\"" Jul 16 21:53:16.933726 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:16.933681 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 16 21:53:16.934470 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:16.934448 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdqfc5"] Jul 16 21:53:17.085679 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:17.085649 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0ac3000e-ac4a-44e2-94ec-d3b056092636-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdqfc5\" (UID: \"0ac3000e-ac4a-44e2-94ec-d3b056092636\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdqfc5" Jul 16 21:53:17.085827 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:17.085691 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7kdd\" (UniqueName: \"kubernetes.io/projected/0ac3000e-ac4a-44e2-94ec-d3b056092636-kube-api-access-n7kdd\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdqfc5\" (UID: \"0ac3000e-ac4a-44e2-94ec-d3b056092636\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdqfc5" Jul 16 21:53:17.085827 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:17.085732 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0ac3000e-ac4a-44e2-94ec-d3b056092636-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdqfc5\" (UID: \"0ac3000e-ac4a-44e2-94ec-d3b056092636\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdqfc5" Jul 16 21:53:17.101769 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:17.101738 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:17.186314 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:17.186248 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0ac3000e-ac4a-44e2-94ec-d3b056092636-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdqfc5\" (UID: \"0ac3000e-ac4a-44e2-94ec-d3b056092636\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdqfc5" Jul 16 21:53:17.186314 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:17.186285 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7kdd\" (UniqueName: \"kubernetes.io/projected/0ac3000e-ac4a-44e2-94ec-d3b056092636-kube-api-access-n7kdd\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdqfc5\" (UID: \"0ac3000e-ac4a-44e2-94ec-d3b056092636\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdqfc5" Jul 16 21:53:17.186453 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:17.186317 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0ac3000e-ac4a-44e2-94ec-d3b056092636-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdqfc5\" (UID: \"0ac3000e-ac4a-44e2-94ec-d3b056092636\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdqfc5" Jul 16 21:53:17.186703 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:17.186686 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0ac3000e-ac4a-44e2-94ec-d3b056092636-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdqfc5\" (UID: \"0ac3000e-ac4a-44e2-94ec-d3b056092636\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdqfc5" Jul 16 21:53:17.186739 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:17.186709 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0ac3000e-ac4a-44e2-94ec-d3b056092636-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdqfc5\" (UID: \"0ac3000e-ac4a-44e2-94ec-d3b056092636\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdqfc5" Jul 16 21:53:17.194961 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:17.194942 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7kdd\" (UniqueName: \"kubernetes.io/projected/0ac3000e-ac4a-44e2-94ec-d3b056092636-kube-api-access-n7kdd\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdqfc5\" (UID: \"0ac3000e-ac4a-44e2-94ec-d3b056092636\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdqfc5" Jul 16 21:53:17.241890 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:17.241867 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdqfc5" Jul 16 21:53:17.371713 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:17.371685 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdqfc5"] Jul 16 21:53:17.374934 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:53:17.374908 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ac3000e_ac4a_44e2_94ec_d3b056092636.slice/crio-c3bfbd9cd5efd575f227d8a428a9c7497dcc544f73006f11424ddb89ac54809b WatchSource:0}: Error finding container c3bfbd9cd5efd575f227d8a428a9c7497dcc544f73006f11424ddb89ac54809b: Status 404 returned error can't find the container with id c3bfbd9cd5efd575f227d8a428a9c7497dcc544f73006f11424ddb89ac54809b Jul 16 21:53:17.388499 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:17.388471 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdqfc5" event={"ID":"0ac3000e-ac4a-44e2-94ec-d3b056092636","Type":"ContainerStarted","Data":"c3bfbd9cd5efd575f227d8a428a9c7497dcc544f73006f11424ddb89ac54809b"} Jul 16 21:53:18.394390 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:18.394352 2585 generic.go:356] "Generic (PLEG): container finished" podID="0ac3000e-ac4a-44e2-94ec-d3b056092636" containerID="79dad5a9e4e61d10cd2b2354038922469348a3ae48b1eec76bae928e15356807" exitCode=0 Jul 16 21:53:18.394801 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:18.394418 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdqfc5" event={"ID":"0ac3000e-ac4a-44e2-94ec-d3b056092636","Type":"ContainerDied","Data":"79dad5a9e4e61d10cd2b2354038922469348a3ae48b1eec76bae928e15356807"} Jul 16 21:53:19.400082 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:19.400051 2585 generic.go:356] "Generic (PLEG): container finished" podID="0ac3000e-ac4a-44e2-94ec-d3b056092636" containerID="c93f786f8d0abad47be5118945df50edab6acfe83d158bb222bb618237c29765" exitCode=0 Jul 16 21:53:19.400462 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:19.400144 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdqfc5" event={"ID":"0ac3000e-ac4a-44e2-94ec-d3b056092636","Type":"ContainerDied","Data":"c93f786f8d0abad47be5118945df50edab6acfe83d158bb222bb618237c29765"} Jul 16 21:53:20.407386 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:20.407331 2585 generic.go:356] "Generic (PLEG): container finished" podID="0ac3000e-ac4a-44e2-94ec-d3b056092636" containerID="9b28cea4566d8d651cab3d470c403097341d69076296089283287f9fb1886c0c" exitCode=0 Jul 16 21:53:20.407796 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:20.407423 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdqfc5" event={"ID":"0ac3000e-ac4a-44e2-94ec-d3b056092636","Type":"ContainerDied","Data":"9b28cea4566d8d651cab3d470c403097341d69076296089283287f9fb1886c0c"} Jul 16 21:53:20.655637 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:20.655602 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:20.723361 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:20.723289 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:21.562180 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:21.562158 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdqfc5" Jul 16 21:53:21.722433 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:21.722352 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0ac3000e-ac4a-44e2-94ec-d3b056092636-bundle\") pod \"0ac3000e-ac4a-44e2-94ec-d3b056092636\" (UID: \"0ac3000e-ac4a-44e2-94ec-d3b056092636\") " Jul 16 21:53:21.722433 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:21.722409 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0ac3000e-ac4a-44e2-94ec-d3b056092636-util\") pod \"0ac3000e-ac4a-44e2-94ec-d3b056092636\" (UID: \"0ac3000e-ac4a-44e2-94ec-d3b056092636\") " Jul 16 21:53:21.722652 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:21.722479 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7kdd\" (UniqueName: \"kubernetes.io/projected/0ac3000e-ac4a-44e2-94ec-d3b056092636-kube-api-access-n7kdd\") pod \"0ac3000e-ac4a-44e2-94ec-d3b056092636\" (UID: \"0ac3000e-ac4a-44e2-94ec-d3b056092636\") " Jul 16 21:53:21.723103 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:21.723078 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ac3000e-ac4a-44e2-94ec-d3b056092636-bundle" (OuterVolumeSpecName: "bundle") pod "0ac3000e-ac4a-44e2-94ec-d3b056092636" (UID: "0ac3000e-ac4a-44e2-94ec-d3b056092636"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:53:21.724525 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:21.724497 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ac3000e-ac4a-44e2-94ec-d3b056092636-kube-api-access-n7kdd" (OuterVolumeSpecName: "kube-api-access-n7kdd") pod "0ac3000e-ac4a-44e2-94ec-d3b056092636" (UID: "0ac3000e-ac4a-44e2-94ec-d3b056092636"). InnerVolumeSpecName "kube-api-access-n7kdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:53:21.728203 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:21.728177 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ac3000e-ac4a-44e2-94ec-d3b056092636-util" (OuterVolumeSpecName: "util") pod "0ac3000e-ac4a-44e2-94ec-d3b056092636" (UID: "0ac3000e-ac4a-44e2-94ec-d3b056092636"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:53:21.823327 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:21.823297 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7kdd\" (UniqueName: \"kubernetes.io/projected/0ac3000e-ac4a-44e2-94ec-d3b056092636-kube-api-access-n7kdd\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:53:21.823327 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:21.823325 2585 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0ac3000e-ac4a-44e2-94ec-d3b056092636-bundle\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:53:21.823470 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:21.823333 2585 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0ac3000e-ac4a-44e2-94ec-d3b056092636-util\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:53:22.418267 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:22.418239 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdqfc5" Jul 16 21:53:22.418267 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:22.418244 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdqfc5" event={"ID":"0ac3000e-ac4a-44e2-94ec-d3b056092636","Type":"ContainerDied","Data":"c3bfbd9cd5efd575f227d8a428a9c7497dcc544f73006f11424ddb89ac54809b"} Jul 16 21:53:22.418469 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:22.418294 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3bfbd9cd5efd575f227d8a428a9c7497dcc544f73006f11424ddb89ac54809b" Jul 16 21:53:33.231166 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:33.231128 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:33.326385 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:33.326341 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:33.850748 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:33.850717 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:34.319113 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:34.319085 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:34.424812 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:34.424781 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:34.465922 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:34.465897 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-574666dc8f-7dmxp"] Jul 16 21:53:34.466254 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:34.466239 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ac3000e-ac4a-44e2-94ec-d3b056092636" containerName="extract" Jul 16 21:53:34.466307 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:34.466257 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ac3000e-ac4a-44e2-94ec-d3b056092636" containerName="extract" Jul 16 21:53:34.466307 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:34.466272 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ac3000e-ac4a-44e2-94ec-d3b056092636" containerName="pull" Jul 16 21:53:34.466307 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:34.466277 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ac3000e-ac4a-44e2-94ec-d3b056092636" containerName="pull" Jul 16 21:53:34.466307 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:34.466288 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ac3000e-ac4a-44e2-94ec-d3b056092636" containerName="util" Jul 16 21:53:34.466307 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:34.466294 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ac3000e-ac4a-44e2-94ec-d3b056092636" containerName="util" Jul 16 21:53:34.466457 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:34.466360 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ac3000e-ac4a-44e2-94ec-d3b056092636" containerName="extract" Jul 16 21:53:34.473063 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:34.473040 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-7dmxp" Jul 16 21:53:34.477111 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:34.477089 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-9vl5d\"" Jul 16 21:53:34.477210 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:34.477146 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 16 21:53:34.477361 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:34.477345 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 16 21:53:34.477779 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:34.477764 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 16 21:53:34.478233 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:34.478197 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 16 21:53:34.511354 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:34.511331 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-574666dc8f-7dmxp"] Jul 16 21:53:34.516726 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:34.516701 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f84b0d43-3571-419d-8232-2f7418f2da3a-manifests\") pod \"opendatahub-operator-controller-manager-574666dc8f-7dmxp\" (UID: \"f84b0d43-3571-419d-8232-2f7418f2da3a\") " pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-7dmxp" Jul 16 21:53:34.516845 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:34.516730 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f84b0d43-3571-419d-8232-2f7418f2da3a-webhook-cert\") pod \"opendatahub-operator-controller-manager-574666dc8f-7dmxp\" (UID: \"f84b0d43-3571-419d-8232-2f7418f2da3a\") " pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-7dmxp" Jul 16 21:53:34.516845 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:34.516806 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f84b0d43-3571-419d-8232-2f7418f2da3a-tmp\") pod \"opendatahub-operator-controller-manager-574666dc8f-7dmxp\" (UID: \"f84b0d43-3571-419d-8232-2f7418f2da3a\") " pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-7dmxp" Jul 16 21:53:34.516845 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:34.516824 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f84b0d43-3571-419d-8232-2f7418f2da3a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-574666dc8f-7dmxp\" (UID: \"f84b0d43-3571-419d-8232-2f7418f2da3a\") " pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-7dmxp" Jul 16 21:53:34.517029 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:34.516896 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hllh\" (UniqueName: \"kubernetes.io/projected/f84b0d43-3571-419d-8232-2f7418f2da3a-kube-api-access-7hllh\") pod \"opendatahub-operator-controller-manager-574666dc8f-7dmxp\" (UID: \"f84b0d43-3571-419d-8232-2f7418f2da3a\") " pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-7dmxp" Jul 16 21:53:34.618197 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:34.618104 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f84b0d43-3571-419d-8232-2f7418f2da3a-tmp\") pod \"opendatahub-operator-controller-manager-574666dc8f-7dmxp\" (UID: \"f84b0d43-3571-419d-8232-2f7418f2da3a\") " pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-7dmxp" Jul 16 21:53:34.618197 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:34.618156 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f84b0d43-3571-419d-8232-2f7418f2da3a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-574666dc8f-7dmxp\" (UID: \"f84b0d43-3571-419d-8232-2f7418f2da3a\") " pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-7dmxp" Jul 16 21:53:34.618448 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:34.618212 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hllh\" (UniqueName: \"kubernetes.io/projected/f84b0d43-3571-419d-8232-2f7418f2da3a-kube-api-access-7hllh\") pod \"opendatahub-operator-controller-manager-574666dc8f-7dmxp\" (UID: \"f84b0d43-3571-419d-8232-2f7418f2da3a\") " pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-7dmxp" Jul 16 21:53:34.618448 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:34.618269 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f84b0d43-3571-419d-8232-2f7418f2da3a-manifests\") pod \"opendatahub-operator-controller-manager-574666dc8f-7dmxp\" (UID: \"f84b0d43-3571-419d-8232-2f7418f2da3a\") " pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-7dmxp" Jul 16 21:53:34.618448 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:34.618294 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f84b0d43-3571-419d-8232-2f7418f2da3a-webhook-cert\") pod \"opendatahub-operator-controller-manager-574666dc8f-7dmxp\" (UID: \"f84b0d43-3571-419d-8232-2f7418f2da3a\") " pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-7dmxp" Jul 16 21:53:34.618768 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:34.618737 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f84b0d43-3571-419d-8232-2f7418f2da3a-manifests\") pod \"opendatahub-operator-controller-manager-574666dc8f-7dmxp\" (UID: \"f84b0d43-3571-419d-8232-2f7418f2da3a\") " pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-7dmxp" Jul 16 21:53:34.621242 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:34.621201 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f84b0d43-3571-419d-8232-2f7418f2da3a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-574666dc8f-7dmxp\" (UID: \"f84b0d43-3571-419d-8232-2f7418f2da3a\") " pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-7dmxp" Jul 16 21:53:34.621731 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:34.621535 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f84b0d43-3571-419d-8232-2f7418f2da3a-tmp\") pod \"opendatahub-operator-controller-manager-574666dc8f-7dmxp\" (UID: \"f84b0d43-3571-419d-8232-2f7418f2da3a\") " pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-7dmxp" Jul 16 21:53:34.622820 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:34.622781 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f84b0d43-3571-419d-8232-2f7418f2da3a-webhook-cert\") pod \"opendatahub-operator-controller-manager-574666dc8f-7dmxp\" (UID: \"f84b0d43-3571-419d-8232-2f7418f2da3a\") " pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-7dmxp" Jul 16 21:53:34.647236 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:34.647196 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hllh\" (UniqueName: \"kubernetes.io/projected/f84b0d43-3571-419d-8232-2f7418f2da3a-kube-api-access-7hllh\") pod \"opendatahub-operator-controller-manager-574666dc8f-7dmxp\" (UID: \"f84b0d43-3571-419d-8232-2f7418f2da3a\") " pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-7dmxp" Jul 16 21:53:34.741387 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:34.741353 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:34.783796 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:34.783766 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-7dmxp" Jul 16 21:53:34.847378 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:34.847349 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:34.950382 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:34.950343 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-574666dc8f-7dmxp"] Jul 16 21:53:34.954233 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:53:34.954196 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf84b0d43_3571_419d_8232_2f7418f2da3a.slice/crio-68f6bfc2cfc087f1f870b81d0e66e9ce09fc3321084fd2b6919e0b074f233a95 WatchSource:0}: Error finding container 68f6bfc2cfc087f1f870b81d0e66e9ce09fc3321084fd2b6919e0b074f233a95: Status 404 returned error can't find the container with id 68f6bfc2cfc087f1f870b81d0e66e9ce09fc3321084fd2b6919e0b074f233a95 Jul 16 21:53:34.955856 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:34.955833 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:35.049533 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:35.049499 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:35.232740 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:35.232668 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:35.323423 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:35.323393 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:35.427621 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:35.427557 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:35.478181 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:35.478140 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-7dmxp" event={"ID":"f84b0d43-3571-419d-8232-2f7418f2da3a","Type":"ContainerStarted","Data":"68f6bfc2cfc087f1f870b81d0e66e9ce09fc3321084fd2b6919e0b074f233a95"} Jul 16 21:53:35.525784 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:35.525703 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:35.650297 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:35.650263 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:37.396588 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:37.396527 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:37.894637 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:37.894615 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:38.494675 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:38.494642 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-7dmxp" event={"ID":"f84b0d43-3571-419d-8232-2f7418f2da3a","Type":"ContainerStarted","Data":"03c537d1a585b7f772a2938f8a01abc4bc86c259d54c0b9623fd8be66674ff0d"} Jul 16 21:53:39.500131 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:39.500100 2585 generic.go:356] "Generic (PLEG): container finished" podID="f84b0d43-3571-419d-8232-2f7418f2da3a" containerID="03c537d1a585b7f772a2938f8a01abc4bc86c259d54c0b9623fd8be66674ff0d" exitCode=0 Jul 16 21:53:39.500508 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:39.500179 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-7dmxp" event={"ID":"f84b0d43-3571-419d-8232-2f7418f2da3a","Type":"ContainerDied","Data":"03c537d1a585b7f772a2938f8a01abc4bc86c259d54c0b9623fd8be66674ff0d"} Jul 16 21:53:40.506404 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:40.506363 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-7dmxp" event={"ID":"f84b0d43-3571-419d-8232-2f7418f2da3a","Type":"ContainerStarted","Data":"cebd964314eddd0efcda12342cad514bd6df909f6986dcbe7222b15f51d1995e"} Jul 16 21:53:40.506855 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:40.506503 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-7dmxp" Jul 16 21:53:40.537049 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:40.537003 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-7dmxp" podStartSLOduration=2.205636151 podStartE2EDuration="6.536990957s" totalImagesPullingTime="3.309299336s" totalInitContainerRuntime="1.02205547s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:53:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:53:40.53361931 +0000 UTC m=+369.156062264" watchObservedRunningTime="2026-07-16 21:53:40.536990957 +0000 UTC m=+369.159433909" Jul 16 21:53:46.152716 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:46.152681 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:46.285697 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:46.285649 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:51.512809 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:51.512782 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-7dmxp" Jul 16 21:53:53.442509 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:53.442468 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:53.727621 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:53.727524 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:54.312053 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:54.312025 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:54.585834 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:54.585744 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-54f57477db-cw8lx"] Jul 16 21:53:54.589111 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:54.589092 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54f57477db-cw8lx" Jul 16 21:53:54.594969 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:54.594948 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-bdbwp\"" Jul 16 21:53:54.595165 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:54.594996 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 16 21:53:54.595787 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:54.595620 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:54.595878 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:54.595855 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 16 21:53:54.598881 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:54.598864 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 16 21:53:54.603546 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:54.603530 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 16 21:53:54.624360 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:54.624333 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54f57477db-cw8lx"] Jul 16 21:53:54.679156 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:54.679124 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2db26749-c5a9-4647-87f0-de77f6ff2f3f-tls-certs\") pod \"kube-auth-proxy-54f57477db-cw8lx\" (UID: \"2db26749-c5a9-4647-87f0-de77f6ff2f3f\") " pod="openshift-ingress/kube-auth-proxy-54f57477db-cw8lx" Jul 16 21:53:54.679307 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:54.679187 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2db26749-c5a9-4647-87f0-de77f6ff2f3f-tmp\") pod \"kube-auth-proxy-54f57477db-cw8lx\" (UID: \"2db26749-c5a9-4647-87f0-de77f6ff2f3f\") " pod="openshift-ingress/kube-auth-proxy-54f57477db-cw8lx" Jul 16 21:53:54.679307 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:54.679272 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57t29\" (UniqueName: \"kubernetes.io/projected/2db26749-c5a9-4647-87f0-de77f6ff2f3f-kube-api-access-57t29\") pod \"kube-auth-proxy-54f57477db-cw8lx\" (UID: \"2db26749-c5a9-4647-87f0-de77f6ff2f3f\") " pod="openshift-ingress/kube-auth-proxy-54f57477db-cw8lx" Jul 16 21:53:54.779775 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:54.779739 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2db26749-c5a9-4647-87f0-de77f6ff2f3f-tls-certs\") pod \"kube-auth-proxy-54f57477db-cw8lx\" (UID: \"2db26749-c5a9-4647-87f0-de77f6ff2f3f\") " pod="openshift-ingress/kube-auth-proxy-54f57477db-cw8lx" Jul 16 21:53:54.779932 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:54.779799 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2db26749-c5a9-4647-87f0-de77f6ff2f3f-tmp\") pod \"kube-auth-proxy-54f57477db-cw8lx\" (UID: \"2db26749-c5a9-4647-87f0-de77f6ff2f3f\") " pod="openshift-ingress/kube-auth-proxy-54f57477db-cw8lx" Jul 16 21:53:54.779932 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:54.779850 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57t29\" (UniqueName: \"kubernetes.io/projected/2db26749-c5a9-4647-87f0-de77f6ff2f3f-kube-api-access-57t29\") pod \"kube-auth-proxy-54f57477db-cw8lx\" (UID: \"2db26749-c5a9-4647-87f0-de77f6ff2f3f\") " pod="openshift-ingress/kube-auth-proxy-54f57477db-cw8lx" Jul 16 21:53:54.782034 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:54.782011 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2db26749-c5a9-4647-87f0-de77f6ff2f3f-tmp\") pod \"kube-auth-proxy-54f57477db-cw8lx\" (UID: \"2db26749-c5a9-4647-87f0-de77f6ff2f3f\") " pod="openshift-ingress/kube-auth-proxy-54f57477db-cw8lx" Jul 16 21:53:54.782278 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:54.782262 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2db26749-c5a9-4647-87f0-de77f6ff2f3f-tls-certs\") pod \"kube-auth-proxy-54f57477db-cw8lx\" (UID: \"2db26749-c5a9-4647-87f0-de77f6ff2f3f\") " pod="openshift-ingress/kube-auth-proxy-54f57477db-cw8lx" Jul 16 21:53:54.795867 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:54.795835 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57t29\" (UniqueName: \"kubernetes.io/projected/2db26749-c5a9-4647-87f0-de77f6ff2f3f-kube-api-access-57t29\") pod \"kube-auth-proxy-54f57477db-cw8lx\" (UID: \"2db26749-c5a9-4647-87f0-de77f6ff2f3f\") " pod="openshift-ingress/kube-auth-proxy-54f57477db-cw8lx" Jul 16 21:53:54.844571 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:54.844484 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:54.898392 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:54.898365 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54f57477db-cw8lx" Jul 16 21:53:55.055379 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:55.055346 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:55.074388 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:55.074367 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54f57477db-cw8lx"] Jul 16 21:53:55.076909 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:53:55.076881 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2db26749_c5a9_4647_87f0_de77f6ff2f3f.slice/crio-55699763ab8ff4fb435dd165c5ea4171679c3f35bbf7e3496643cc6f4525d671 WatchSource:0}: Error finding container 55699763ab8ff4fb435dd165c5ea4171679c3f35bbf7e3496643cc6f4525d671: Status 404 returned error can't find the container with id 55699763ab8ff4fb435dd165c5ea4171679c3f35bbf7e3496643cc6f4525d671 Jul 16 21:53:55.221484 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:55.221409 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:55.579287 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:55.579147 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54f57477db-cw8lx" event={"ID":"2db26749-c5a9-4647-87f0-de77f6ff2f3f","Type":"ContainerStarted","Data":"55699763ab8ff4fb435dd165c5ea4171679c3f35bbf7e3496643cc6f4525d671"} Jul 16 21:53:56.037639 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:56.037555 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:56.056199 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:56.056159 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-xg9kf"] Jul 16 21:53:56.059602 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:56.059579 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-xg9kf" Jul 16 21:53:56.062069 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:56.062040 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 16 21:53:56.062278 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:56.062256 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-bz75p\"" Jul 16 21:53:56.080342 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:56.080316 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-xg9kf"] Jul 16 21:53:56.190762 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:56.190730 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40d4f474-fbc3-4196-bf87-8a3510483609-tls-certs\") pod \"model-serving-api-54d89bd5fc-xg9kf\" (UID: \"40d4f474-fbc3-4196-bf87-8a3510483609\") " pod="opendatahub/model-serving-api-54d89bd5fc-xg9kf" Jul 16 21:53:56.190926 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:56.190781 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csr42\" (UniqueName: \"kubernetes.io/projected/40d4f474-fbc3-4196-bf87-8a3510483609-kube-api-access-csr42\") pod \"model-serving-api-54d89bd5fc-xg9kf\" (UID: \"40d4f474-fbc3-4196-bf87-8a3510483609\") " pod="opendatahub/model-serving-api-54d89bd5fc-xg9kf" Jul 16 21:53:56.282636 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:56.282604 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:56.291819 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:56.291753 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40d4f474-fbc3-4196-bf87-8a3510483609-tls-certs\") pod \"model-serving-api-54d89bd5fc-xg9kf\" (UID: \"40d4f474-fbc3-4196-bf87-8a3510483609\") " pod="opendatahub/model-serving-api-54d89bd5fc-xg9kf" Jul 16 21:53:56.292355 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:56.292324 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csr42\" (UniqueName: \"kubernetes.io/projected/40d4f474-fbc3-4196-bf87-8a3510483609-kube-api-access-csr42\") pod \"model-serving-api-54d89bd5fc-xg9kf\" (UID: \"40d4f474-fbc3-4196-bf87-8a3510483609\") " pod="opendatahub/model-serving-api-54d89bd5fc-xg9kf" Jul 16 21:53:56.296769 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:56.295975 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40d4f474-fbc3-4196-bf87-8a3510483609-tls-certs\") pod \"model-serving-api-54d89bd5fc-xg9kf\" (UID: \"40d4f474-fbc3-4196-bf87-8a3510483609\") " pod="opendatahub/model-serving-api-54d89bd5fc-xg9kf" Jul 16 21:53:56.305550 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:56.305511 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csr42\" (UniqueName: \"kubernetes.io/projected/40d4f474-fbc3-4196-bf87-8a3510483609-kube-api-access-csr42\") pod \"model-serving-api-54d89bd5fc-xg9kf\" (UID: \"40d4f474-fbc3-4196-bf87-8a3510483609\") " pod="opendatahub/model-serving-api-54d89bd5fc-xg9kf" Jul 16 21:53:56.373535 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:56.373501 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-xg9kf" Jul 16 21:53:56.523475 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:56.523438 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-xg9kf"] Jul 16 21:53:56.528825 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:53:56.528801 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40d4f474_fbc3_4196_bf87_8a3510483609.slice/crio-fee063111c6bc5f3698190f7ebff4737ffe9b9854a3a3d21a3987e745c5b1a30 WatchSource:0}: Error finding container fee063111c6bc5f3698190f7ebff4737ffe9b9854a3a3d21a3987e745c5b1a30: Status 404 returned error can't find the container with id fee063111c6bc5f3698190f7ebff4737ffe9b9854a3a3d21a3987e745c5b1a30 Jul 16 21:53:56.586716 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:56.586682 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-xg9kf" event={"ID":"40d4f474-fbc3-4196-bf87-8a3510483609","Type":"ContainerStarted","Data":"fee063111c6bc5f3698190f7ebff4737ffe9b9854a3a3d21a3987e745c5b1a30"} Jul 16 21:53:57.039100 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:57.039024 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:57.157971 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:57.157939 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:58.449393 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:58.449350 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:53:58.604027 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:58.603966 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54f57477db-cw8lx" event={"ID":"2db26749-c5a9-4647-87f0-de77f6ff2f3f","Type":"ContainerStarted","Data":"89e8861f141aef9cb889827a689ae38b23e8c1843841ea45bc5df2c9461f6bdb"} Jul 16 21:53:58.606654 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:58.606629 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-xg9kf" event={"ID":"40d4f474-fbc3-4196-bf87-8a3510483609","Type":"ContainerStarted","Data":"d6b44456ee50b698fc01e8c6946649aa3630d5e3d8a3068b568da42c1a40544e"} Jul 16 21:53:58.606797 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:58.606769 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-xg9kf" Jul 16 21:53:58.627994 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:58.627949 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-54f57477db-cw8lx" podStartSLOduration=2.127280161 podStartE2EDuration="4.627935578s" totalImagesPullingTime="2.500655417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:53:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:53:58.627558696 +0000 UTC m=+387.250001650" watchObservedRunningTime="2026-07-16 21:53:58.627935578 +0000 UTC m=+387.250378531" Jul 16 21:53:58.663963 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:53:58.663913 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-xg9kf" podStartSLOduration=0.749001877 podStartE2EDuration="2.663893509s" totalImagesPullingTime="1.914891632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:53:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:53:58.662249769 +0000 UTC m=+387.284692722" watchObservedRunningTime="2026-07-16 21:53:58.663893509 +0000 UTC m=+387.286336461" Jul 16 21:54:02.816519 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:02.816484 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6bfbbdbb89-9spw5"] Jul 16 21:54:02.819935 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:02.819917 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6bfbbdbb89-9spw5" Jul 16 21:54:02.823204 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:02.823178 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 16 21:54:02.824189 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:02.824172 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 16 21:54:02.824279 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:02.824172 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 16 21:54:02.824451 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:02.824434 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-qq7v5\"" Jul 16 21:54:02.824529 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:02.824513 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 16 21:54:02.824574 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:02.824514 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 16 21:54:02.841880 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:02.841851 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6bfbbdbb89-9spw5"] Jul 16 21:54:02.890342 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:02.890310 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:54:02.952361 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:02.952322 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ba61693-5313-4873-b93b-ee97d158d3e9-cert\") pod \"lws-controller-manager-6bfbbdbb89-9spw5\" (UID: \"1ba61693-5313-4873-b93b-ee97d158d3e9\") " pod="openshift-lws-operator/lws-controller-manager-6bfbbdbb89-9spw5" Jul 16 21:54:02.952545 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:02.952377 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1ba61693-5313-4873-b93b-ee97d158d3e9-metrics-cert\") pod \"lws-controller-manager-6bfbbdbb89-9spw5\" (UID: \"1ba61693-5313-4873-b93b-ee97d158d3e9\") " pod="openshift-lws-operator/lws-controller-manager-6bfbbdbb89-9spw5" Jul 16 21:54:02.952545 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:02.952432 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1ba61693-5313-4873-b93b-ee97d158d3e9-manager-config\") pod \"lws-controller-manager-6bfbbdbb89-9spw5\" (UID: \"1ba61693-5313-4873-b93b-ee97d158d3e9\") " pod="openshift-lws-operator/lws-controller-manager-6bfbbdbb89-9spw5" Jul 16 21:54:02.952662 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:02.952552 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qsdj\" (UniqueName: \"kubernetes.io/projected/1ba61693-5313-4873-b93b-ee97d158d3e9-kube-api-access-2qsdj\") pod \"lws-controller-manager-6bfbbdbb89-9spw5\" (UID: \"1ba61693-5313-4873-b93b-ee97d158d3e9\") " pod="openshift-lws-operator/lws-controller-manager-6bfbbdbb89-9spw5" Jul 16 21:54:03.053125 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:03.053086 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1ba61693-5313-4873-b93b-ee97d158d3e9-manager-config\") pod \"lws-controller-manager-6bfbbdbb89-9spw5\" (UID: \"1ba61693-5313-4873-b93b-ee97d158d3e9\") " pod="openshift-lws-operator/lws-controller-manager-6bfbbdbb89-9spw5" Jul 16 21:54:03.053354 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:03.053147 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qsdj\" (UniqueName: \"kubernetes.io/projected/1ba61693-5313-4873-b93b-ee97d158d3e9-kube-api-access-2qsdj\") pod \"lws-controller-manager-6bfbbdbb89-9spw5\" (UID: \"1ba61693-5313-4873-b93b-ee97d158d3e9\") " pod="openshift-lws-operator/lws-controller-manager-6bfbbdbb89-9spw5" Jul 16 21:54:03.053354 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:03.053177 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ba61693-5313-4873-b93b-ee97d158d3e9-cert\") pod \"lws-controller-manager-6bfbbdbb89-9spw5\" (UID: \"1ba61693-5313-4873-b93b-ee97d158d3e9\") " pod="openshift-lws-operator/lws-controller-manager-6bfbbdbb89-9spw5" Jul 16 21:54:03.053354 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:03.053199 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1ba61693-5313-4873-b93b-ee97d158d3e9-metrics-cert\") pod \"lws-controller-manager-6bfbbdbb89-9spw5\" (UID: \"1ba61693-5313-4873-b93b-ee97d158d3e9\") " pod="openshift-lws-operator/lws-controller-manager-6bfbbdbb89-9spw5" Jul 16 21:54:03.053758 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:03.053737 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1ba61693-5313-4873-b93b-ee97d158d3e9-manager-config\") pod \"lws-controller-manager-6bfbbdbb89-9spw5\" (UID: \"1ba61693-5313-4873-b93b-ee97d158d3e9\") " pod="openshift-lws-operator/lws-controller-manager-6bfbbdbb89-9spw5" Jul 16 21:54:03.055739 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:03.055711 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1ba61693-5313-4873-b93b-ee97d158d3e9-metrics-cert\") pod \"lws-controller-manager-6bfbbdbb89-9spw5\" (UID: \"1ba61693-5313-4873-b93b-ee97d158d3e9\") " pod="openshift-lws-operator/lws-controller-manager-6bfbbdbb89-9spw5" Jul 16 21:54:03.055857 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:03.055770 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ba61693-5313-4873-b93b-ee97d158d3e9-cert\") pod \"lws-controller-manager-6bfbbdbb89-9spw5\" (UID: \"1ba61693-5313-4873-b93b-ee97d158d3e9\") " pod="openshift-lws-operator/lws-controller-manager-6bfbbdbb89-9spw5" Jul 16 21:54:03.067401 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:03.067338 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qsdj\" (UniqueName: \"kubernetes.io/projected/1ba61693-5313-4873-b93b-ee97d158d3e9-kube-api-access-2qsdj\") pod \"lws-controller-manager-6bfbbdbb89-9spw5\" (UID: \"1ba61693-5313-4873-b93b-ee97d158d3e9\") " pod="openshift-lws-operator/lws-controller-manager-6bfbbdbb89-9spw5" Jul 16 21:54:03.073452 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:03.073430 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:54:03.129528 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:03.129505 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6bfbbdbb89-9spw5" Jul 16 21:54:03.323615 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:03.323589 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6bfbbdbb89-9spw5"] Jul 16 21:54:03.326151 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:54:03.326121 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ba61693_5313_4873_b93b_ee97d158d3e9.slice/crio-4b12375f1ad9f89b1c97854230faae5a8dc7c332b1637b8eb944df28b338fe87 WatchSource:0}: Error finding container 4b12375f1ad9f89b1c97854230faae5a8dc7c332b1637b8eb944df28b338fe87: Status 404 returned error can't find the container with id 4b12375f1ad9f89b1c97854230faae5a8dc7c332b1637b8eb944df28b338fe87 Jul 16 21:54:03.630213 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:03.630117 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6bfbbdbb89-9spw5" event={"ID":"1ba61693-5313-4873-b93b-ee97d158d3e9","Type":"ContainerStarted","Data":"4b12375f1ad9f89b1c97854230faae5a8dc7c332b1637b8eb944df28b338fe87"} Jul 16 21:54:04.447323 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:04.447275 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:54:04.758546 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:04.758500 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:54:05.004424 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:05.004349 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:54:05.643079 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:05.643045 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6bfbbdbb89-9spw5" event={"ID":"1ba61693-5313-4873-b93b-ee97d158d3e9","Type":"ContainerStarted","Data":"51fcff20d813cc00ecee22384aaf44552fabae6349e4c4fa3712312ad8c1f66b"} Jul 16 21:54:05.643461 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:05.643156 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6bfbbdbb89-9spw5" Jul 16 21:54:05.665735 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:05.665689 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6bfbbdbb89-9spw5" podStartSLOduration=2.359866766 podStartE2EDuration="3.665673235s" totalImagesPullingTime="1.305806469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:54:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:54:05.663830428 +0000 UTC m=+394.286273382" watchObservedRunningTime="2026-07-16 21:54:05.665673235 +0000 UTC m=+394.288116188" Jul 16 21:54:07.597053 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:07.597009 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:54:07.881620 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:07.881525 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:54:08.004093 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:08.004057 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:54:08.131709 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:08.131631 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:54:08.320329 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:08.320297 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:54:09.614936 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:09.614909 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-xg9kf" Jul 16 21:54:16.650912 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:16.650879 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6bfbbdbb89-9spw5" Jul 16 21:54:22.182465 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:22.182428 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:54:22.285204 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:22.285174 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:54:23.261535 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:23.261502 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:54:54.903838 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:54.903763 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:54:55.021984 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:55.021950 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:54:55.132690 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:55.132648 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:54:55.816167 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:55.816131 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:54:56.024179 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:54:56.024145 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:55:10.608710 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:10.608673 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596njrj"] Jul 16 21:55:10.612096 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:10.612076 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596njrj" Jul 16 21:55:10.615623 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:10.615602 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rbwj4\"" Jul 16 21:55:10.616464 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:10.616439 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 16 21:55:10.616543 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:10.616441 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 16 21:55:10.624201 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:10.624176 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596njrj"] Jul 16 21:55:10.685965 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:10.685927 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8bbdf3c4-c287-4af3-be2c-6af952e8a9c2-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596njrj\" (UID: \"8bbdf3c4-c287-4af3-be2c-6af952e8a9c2\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596njrj" Jul 16 21:55:10.686126 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:10.685973 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z854t\" (UniqueName: \"kubernetes.io/projected/8bbdf3c4-c287-4af3-be2c-6af952e8a9c2-kube-api-access-z854t\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596njrj\" (UID: \"8bbdf3c4-c287-4af3-be2c-6af952e8a9c2\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596njrj" Jul 16 21:55:10.686126 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:10.686048 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8bbdf3c4-c287-4af3-be2c-6af952e8a9c2-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596njrj\" (UID: \"8bbdf3c4-c287-4af3-be2c-6af952e8a9c2\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596njrj" Jul 16 21:55:10.786484 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:10.786446 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8bbdf3c4-c287-4af3-be2c-6af952e8a9c2-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596njrj\" (UID: \"8bbdf3c4-c287-4af3-be2c-6af952e8a9c2\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596njrj" Jul 16 21:55:10.786643 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:10.786498 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z854t\" (UniqueName: \"kubernetes.io/projected/8bbdf3c4-c287-4af3-be2c-6af952e8a9c2-kube-api-access-z854t\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596njrj\" (UID: \"8bbdf3c4-c287-4af3-be2c-6af952e8a9c2\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596njrj" Jul 16 21:55:10.786643 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:10.786531 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8bbdf3c4-c287-4af3-be2c-6af952e8a9c2-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596njrj\" (UID: \"8bbdf3c4-c287-4af3-be2c-6af952e8a9c2\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596njrj" Jul 16 21:55:10.786825 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:10.786807 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8bbdf3c4-c287-4af3-be2c-6af952e8a9c2-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596njrj\" (UID: \"8bbdf3c4-c287-4af3-be2c-6af952e8a9c2\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596njrj" Jul 16 21:55:10.786863 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:10.786836 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8bbdf3c4-c287-4af3-be2c-6af952e8a9c2-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596njrj\" (UID: \"8bbdf3c4-c287-4af3-be2c-6af952e8a9c2\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596njrj" Jul 16 21:55:10.796879 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:10.796858 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z854t\" (UniqueName: \"kubernetes.io/projected/8bbdf3c4-c287-4af3-be2c-6af952e8a9c2-kube-api-access-z854t\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596njrj\" (UID: \"8bbdf3c4-c287-4af3-be2c-6af952e8a9c2\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596njrj" Jul 16 21:55:10.921781 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:10.921698 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596njrj" Jul 16 21:55:11.067097 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:11.067064 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596njrj"] Jul 16 21:55:11.068964 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:55:11.068923 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bbdf3c4_c287_4af3_be2c_6af952e8a9c2.slice/crio-a78ef23ea7636fcbbce929a8212c379748d72a42671d6991f55d712c030a4d2c WatchSource:0}: Error finding container a78ef23ea7636fcbbce929a8212c379748d72a42671d6991f55d712c030a4d2c: Status 404 returned error can't find the container with id a78ef23ea7636fcbbce929a8212c379748d72a42671d6991f55d712c030a4d2c Jul 16 21:55:11.197343 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:11.197267 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739qtkh"] Jul 16 21:55:11.200578 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:11.200562 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739qtkh" Jul 16 21:55:11.208985 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:11.208962 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739qtkh"] Jul 16 21:55:11.292039 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:11.292005 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7b6c3b0d-c66b-46ee-b013-b5bb9e619888-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739qtkh\" (UID: \"7b6c3b0d-c66b-46ee-b013-b5bb9e619888\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739qtkh" Jul 16 21:55:11.292174 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:11.292045 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7b6c3b0d-c66b-46ee-b013-b5bb9e619888-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739qtkh\" (UID: \"7b6c3b0d-c66b-46ee-b013-b5bb9e619888\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739qtkh" Jul 16 21:55:11.292174 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:11.292104 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lczf\" (UniqueName: \"kubernetes.io/projected/7b6c3b0d-c66b-46ee-b013-b5bb9e619888-kube-api-access-5lczf\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739qtkh\" (UID: \"7b6c3b0d-c66b-46ee-b013-b5bb9e619888\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739qtkh" Jul 16 21:55:11.393369 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:11.393333 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7b6c3b0d-c66b-46ee-b013-b5bb9e619888-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739qtkh\" (UID: \"7b6c3b0d-c66b-46ee-b013-b5bb9e619888\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739qtkh" Jul 16 21:55:11.393576 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:11.393389 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lczf\" (UniqueName: \"kubernetes.io/projected/7b6c3b0d-c66b-46ee-b013-b5bb9e619888-kube-api-access-5lczf\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739qtkh\" (UID: \"7b6c3b0d-c66b-46ee-b013-b5bb9e619888\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739qtkh" Jul 16 21:55:11.393576 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:11.393451 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7b6c3b0d-c66b-46ee-b013-b5bb9e619888-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739qtkh\" (UID: \"7b6c3b0d-c66b-46ee-b013-b5bb9e619888\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739qtkh" Jul 16 21:55:11.393786 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:11.393762 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7b6c3b0d-c66b-46ee-b013-b5bb9e619888-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739qtkh\" (UID: \"7b6c3b0d-c66b-46ee-b013-b5bb9e619888\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739qtkh" Jul 16 21:55:11.393844 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:11.393790 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7b6c3b0d-c66b-46ee-b013-b5bb9e619888-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739qtkh\" (UID: \"7b6c3b0d-c66b-46ee-b013-b5bb9e619888\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739qtkh" Jul 16 21:55:11.401773 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:11.401750 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lczf\" (UniqueName: \"kubernetes.io/projected/7b6c3b0d-c66b-46ee-b013-b5bb9e619888-kube-api-access-5lczf\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739qtkh\" (UID: \"7b6c3b0d-c66b-46ee-b013-b5bb9e619888\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739qtkh" Jul 16 21:55:11.546856 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:11.546829 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739qtkh" Jul 16 21:55:11.677036 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:11.677009 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739qtkh"] Jul 16 21:55:11.678451 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:55:11.678422 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b6c3b0d_c66b_46ee_b013_b5bb9e619888.slice/crio-e64abe77efe8ff746d90ecfb9ebee081dab1dd2d77edb68cee81b7099e0bca0c WatchSource:0}: Error finding container e64abe77efe8ff746d90ecfb9ebee081dab1dd2d77edb68cee81b7099e0bca0c: Status 404 returned error can't find the container with id e64abe77efe8ff746d90ecfb9ebee081dab1dd2d77edb68cee81b7099e0bca0c Jul 16 21:55:11.949986 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:11.949957 2585 generic.go:356] "Generic (PLEG): container finished" podID="7b6c3b0d-c66b-46ee-b013-b5bb9e619888" containerID="caec02023ccef13e9e67afe6cb00fb204b19add0359d04c1248cd6b2b53ffced" exitCode=0 Jul 16 21:55:11.950176 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:11.950046 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739qtkh" event={"ID":"7b6c3b0d-c66b-46ee-b013-b5bb9e619888","Type":"ContainerDied","Data":"caec02023ccef13e9e67afe6cb00fb204b19add0359d04c1248cd6b2b53ffced"} Jul 16 21:55:11.950176 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:11.950098 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739qtkh" event={"ID":"7b6c3b0d-c66b-46ee-b013-b5bb9e619888","Type":"ContainerStarted","Data":"e64abe77efe8ff746d90ecfb9ebee081dab1dd2d77edb68cee81b7099e0bca0c"} Jul 16 21:55:11.951564 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:11.951543 2585 generic.go:356] "Generic (PLEG): container finished" podID="8bbdf3c4-c287-4af3-be2c-6af952e8a9c2" containerID="d6ec9eed9560ec4f4ef8636d1df42647d779db402a56ac456421c268b09f8360" exitCode=0 Jul 16 21:55:11.951664 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:11.951571 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596njrj" event={"ID":"8bbdf3c4-c287-4af3-be2c-6af952e8a9c2","Type":"ContainerDied","Data":"d6ec9eed9560ec4f4ef8636d1df42647d779db402a56ac456421c268b09f8360"} Jul 16 21:55:11.951664 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:11.951619 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596njrj" event={"ID":"8bbdf3c4-c287-4af3-be2c-6af952e8a9c2","Type":"ContainerStarted","Data":"a78ef23ea7636fcbbce929a8212c379748d72a42671d6991f55d712c030a4d2c"} Jul 16 21:55:11.995823 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:11.995789 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fvbjb"] Jul 16 21:55:11.999298 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:11.999278 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fvbjb" Jul 16 21:55:12.009745 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:12.009722 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fvbjb"] Jul 16 21:55:12.099135 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:12.099061 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1deb487d-8753-4dc6-a838-2c20e9841647-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fvbjb\" (UID: \"1deb487d-8753-4dc6-a838-2c20e9841647\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fvbjb" Jul 16 21:55:12.099135 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:12.099117 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1deb487d-8753-4dc6-a838-2c20e9841647-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fvbjb\" (UID: \"1deb487d-8753-4dc6-a838-2c20e9841647\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fvbjb" Jul 16 21:55:12.099334 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:12.099182 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrw5v\" (UniqueName: \"kubernetes.io/projected/1deb487d-8753-4dc6-a838-2c20e9841647-kube-api-access-lrw5v\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fvbjb\" (UID: \"1deb487d-8753-4dc6-a838-2c20e9841647\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fvbjb" Jul 16 21:55:12.200447 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:12.200410 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1deb487d-8753-4dc6-a838-2c20e9841647-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fvbjb\" (UID: \"1deb487d-8753-4dc6-a838-2c20e9841647\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fvbjb" Jul 16 21:55:12.200612 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:12.200454 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1deb487d-8753-4dc6-a838-2c20e9841647-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fvbjb\" (UID: \"1deb487d-8753-4dc6-a838-2c20e9841647\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fvbjb" Jul 16 21:55:12.200612 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:12.200591 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrw5v\" (UniqueName: \"kubernetes.io/projected/1deb487d-8753-4dc6-a838-2c20e9841647-kube-api-access-lrw5v\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fvbjb\" (UID: \"1deb487d-8753-4dc6-a838-2c20e9841647\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fvbjb" Jul 16 21:55:12.200799 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:12.200782 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1deb487d-8753-4dc6-a838-2c20e9841647-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fvbjb\" (UID: \"1deb487d-8753-4dc6-a838-2c20e9841647\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fvbjb" Jul 16 21:55:12.200875 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:12.200859 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1deb487d-8753-4dc6-a838-2c20e9841647-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fvbjb\" (UID: \"1deb487d-8753-4dc6-a838-2c20e9841647\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fvbjb" Jul 16 21:55:12.209452 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:12.209424 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrw5v\" (UniqueName: \"kubernetes.io/projected/1deb487d-8753-4dc6-a838-2c20e9841647-kube-api-access-lrw5v\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fvbjb\" (UID: \"1deb487d-8753-4dc6-a838-2c20e9841647\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fvbjb" Jul 16 21:55:12.308505 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:12.308470 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fvbjb" Jul 16 21:55:12.444409 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:12.444378 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fvbjb"] Jul 16 21:55:12.614267 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:12.614179 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e089f4p"] Jul 16 21:55:12.617749 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:12.617728 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e089f4p" Jul 16 21:55:12.626576 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:12.626549 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e089f4p"] Jul 16 21:55:12.705594 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:12.705559 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc100a65-eb66-4a34-9e07-7fd60fe7d3f5-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e089f4p\" (UID: \"bc100a65-eb66-4a34-9e07-7fd60fe7d3f5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e089f4p" Jul 16 21:55:12.705946 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:12.705610 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc100a65-eb66-4a34-9e07-7fd60fe7d3f5-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e089f4p\" (UID: \"bc100a65-eb66-4a34-9e07-7fd60fe7d3f5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e089f4p" Jul 16 21:55:12.705946 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:12.705638 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjsfz\" (UniqueName: \"kubernetes.io/projected/bc100a65-eb66-4a34-9e07-7fd60fe7d3f5-kube-api-access-cjsfz\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e089f4p\" (UID: \"bc100a65-eb66-4a34-9e07-7fd60fe7d3f5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e089f4p" Jul 16 21:55:12.806313 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:12.806173 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc100a65-eb66-4a34-9e07-7fd60fe7d3f5-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e089f4p\" (UID: \"bc100a65-eb66-4a34-9e07-7fd60fe7d3f5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e089f4p" Jul 16 21:55:12.806313 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:12.806229 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc100a65-eb66-4a34-9e07-7fd60fe7d3f5-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e089f4p\" (UID: \"bc100a65-eb66-4a34-9e07-7fd60fe7d3f5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e089f4p" Jul 16 21:55:12.806313 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:12.806260 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjsfz\" (UniqueName: \"kubernetes.io/projected/bc100a65-eb66-4a34-9e07-7fd60fe7d3f5-kube-api-access-cjsfz\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e089f4p\" (UID: \"bc100a65-eb66-4a34-9e07-7fd60fe7d3f5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e089f4p" Jul 16 21:55:12.806543 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:12.806522 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc100a65-eb66-4a34-9e07-7fd60fe7d3f5-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e089f4p\" (UID: \"bc100a65-eb66-4a34-9e07-7fd60fe7d3f5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e089f4p" Jul 16 21:55:12.806631 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:12.806612 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc100a65-eb66-4a34-9e07-7fd60fe7d3f5-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e089f4p\" (UID: \"bc100a65-eb66-4a34-9e07-7fd60fe7d3f5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e089f4p" Jul 16 21:55:12.814632 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:12.814612 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjsfz\" (UniqueName: \"kubernetes.io/projected/bc100a65-eb66-4a34-9e07-7fd60fe7d3f5-kube-api-access-cjsfz\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e089f4p\" (UID: \"bc100a65-eb66-4a34-9e07-7fd60fe7d3f5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e089f4p" Jul 16 21:55:12.927932 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:12.927860 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e089f4p" Jul 16 21:55:12.958368 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:12.958340 2585 generic.go:356] "Generic (PLEG): container finished" podID="1deb487d-8753-4dc6-a838-2c20e9841647" containerID="3a2a68e3e71df8b66b2440b33686014e490338f7596875cfd8979b93c4d4a694" exitCode=0 Jul 16 21:55:12.958498 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:12.958378 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fvbjb" event={"ID":"1deb487d-8753-4dc6-a838-2c20e9841647","Type":"ContainerDied","Data":"3a2a68e3e71df8b66b2440b33686014e490338f7596875cfd8979b93c4d4a694"} Jul 16 21:55:12.958498 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:12.958446 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fvbjb" event={"ID":"1deb487d-8753-4dc6-a838-2c20e9841647","Type":"ContainerStarted","Data":"63cb49346e32669f8a7c3f822911d71328cb64b23d81045c5152ba18b0d863d8"} Jul 16 21:55:12.960613 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:12.960580 2585 generic.go:356] "Generic (PLEG): container finished" podID="8bbdf3c4-c287-4af3-be2c-6af952e8a9c2" containerID="ca250737e71641cd186a8c678cdbbfb8c3530265a7f0d694c49ce596b5222844" exitCode=0 Jul 16 21:55:12.960717 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:12.960635 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596njrj" event={"ID":"8bbdf3c4-c287-4af3-be2c-6af952e8a9c2","Type":"ContainerDied","Data":"ca250737e71641cd186a8c678cdbbfb8c3530265a7f0d694c49ce596b5222844"} Jul 16 21:55:13.078171 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:13.078133 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e089f4p"] Jul 16 21:55:13.101087 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:55:13.101056 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc100a65_eb66_4a34_9e07_7fd60fe7d3f5.slice/crio-dbc7832b45a8c4988b1888c1d3060d8fc97168b52e5665464b16dc13fe805933 WatchSource:0}: Error finding container dbc7832b45a8c4988b1888c1d3060d8fc97168b52e5665464b16dc13fe805933: Status 404 returned error can't find the container with id dbc7832b45a8c4988b1888c1d3060d8fc97168b52e5665464b16dc13fe805933 Jul 16 21:55:13.235879 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:55:13.235854 2585 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc100a65_eb66_4a34_9e07_7fd60fe7d3f5.slice/crio-32c26ae2e9af120207255858e2b0f236ace3038581d969fbaf149a9819679709.scope\": RecentStats: unable to find data in memory cache]" Jul 16 21:55:13.969642 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:13.969559 2585 generic.go:356] "Generic (PLEG): container finished" podID="7b6c3b0d-c66b-46ee-b013-b5bb9e619888" containerID="31188100cfbeaf5e5132649e993e0092ed02e92dca4472dc250462b6a23520bd" exitCode=0 Jul 16 21:55:13.969642 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:13.969614 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739qtkh" event={"ID":"7b6c3b0d-c66b-46ee-b013-b5bb9e619888","Type":"ContainerDied","Data":"31188100cfbeaf5e5132649e993e0092ed02e92dca4472dc250462b6a23520bd"} Jul 16 21:55:13.971869 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:13.971845 2585 generic.go:356] "Generic (PLEG): container finished" podID="8bbdf3c4-c287-4af3-be2c-6af952e8a9c2" containerID="2de398cbfd01b852d03b78fc8d4eadf34349d10d62f7dfb3219d58de70477ee1" exitCode=0 Jul 16 21:55:13.971969 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:13.971922 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596njrj" event={"ID":"8bbdf3c4-c287-4af3-be2c-6af952e8a9c2","Type":"ContainerDied","Data":"2de398cbfd01b852d03b78fc8d4eadf34349d10d62f7dfb3219d58de70477ee1"} Jul 16 21:55:13.973267 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:13.973245 2585 generic.go:356] "Generic (PLEG): container finished" podID="bc100a65-eb66-4a34-9e07-7fd60fe7d3f5" containerID="32c26ae2e9af120207255858e2b0f236ace3038581d969fbaf149a9819679709" exitCode=0 Jul 16 21:55:13.973351 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:13.973330 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e089f4p" event={"ID":"bc100a65-eb66-4a34-9e07-7fd60fe7d3f5","Type":"ContainerDied","Data":"32c26ae2e9af120207255858e2b0f236ace3038581d969fbaf149a9819679709"} Jul 16 21:55:13.973392 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:13.973369 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e089f4p" event={"ID":"bc100a65-eb66-4a34-9e07-7fd60fe7d3f5","Type":"ContainerStarted","Data":"dbc7832b45a8c4988b1888c1d3060d8fc97168b52e5665464b16dc13fe805933"} Jul 16 21:55:13.975159 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:13.975141 2585 generic.go:356] "Generic (PLEG): container finished" podID="1deb487d-8753-4dc6-a838-2c20e9841647" containerID="ff0c174979174df1b98855de5ca1d71ba465a2e0bf60bdd6211f4c47536fa439" exitCode=0 Jul 16 21:55:13.975238 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:13.975191 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fvbjb" event={"ID":"1deb487d-8753-4dc6-a838-2c20e9841647","Type":"ContainerDied","Data":"ff0c174979174df1b98855de5ca1d71ba465a2e0bf60bdd6211f4c47536fa439"} Jul 16 21:55:14.981639 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:14.981604 2585 generic.go:356] "Generic (PLEG): container finished" podID="7b6c3b0d-c66b-46ee-b013-b5bb9e619888" containerID="28870ebdb7cb0a1dbaa29b7809641d0118e5b79bbeba5abefe90bf4760c16bc6" exitCode=0 Jul 16 21:55:14.982071 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:14.981697 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739qtkh" event={"ID":"7b6c3b0d-c66b-46ee-b013-b5bb9e619888","Type":"ContainerDied","Data":"28870ebdb7cb0a1dbaa29b7809641d0118e5b79bbeba5abefe90bf4760c16bc6"} Jul 16 21:55:14.983497 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:14.983474 2585 generic.go:356] "Generic (PLEG): container finished" podID="bc100a65-eb66-4a34-9e07-7fd60fe7d3f5" containerID="de8b29301bc0aa2d36518af7e9c710cd707cc397f77c1333fa2e21bbacbaa377" exitCode=0 Jul 16 21:55:14.983587 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:14.983552 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e089f4p" event={"ID":"bc100a65-eb66-4a34-9e07-7fd60fe7d3f5","Type":"ContainerDied","Data":"de8b29301bc0aa2d36518af7e9c710cd707cc397f77c1333fa2e21bbacbaa377"} Jul 16 21:55:14.985763 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:14.985738 2585 generic.go:356] "Generic (PLEG): container finished" podID="1deb487d-8753-4dc6-a838-2c20e9841647" containerID="b0a3bd321ba6188c19e6ab7db52a7fd20661e8c56e0217af24a9d3b232523649" exitCode=0 Jul 16 21:55:14.985869 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:14.985824 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fvbjb" event={"ID":"1deb487d-8753-4dc6-a838-2c20e9841647","Type":"ContainerDied","Data":"b0a3bd321ba6188c19e6ab7db52a7fd20661e8c56e0217af24a9d3b232523649"} Jul 16 21:55:15.120549 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:15.120525 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596njrj" Jul 16 21:55:15.227971 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:15.227934 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z854t\" (UniqueName: \"kubernetes.io/projected/8bbdf3c4-c287-4af3-be2c-6af952e8a9c2-kube-api-access-z854t\") pod \"8bbdf3c4-c287-4af3-be2c-6af952e8a9c2\" (UID: \"8bbdf3c4-c287-4af3-be2c-6af952e8a9c2\") " Jul 16 21:55:15.228214 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:15.228036 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8bbdf3c4-c287-4af3-be2c-6af952e8a9c2-bundle\") pod \"8bbdf3c4-c287-4af3-be2c-6af952e8a9c2\" (UID: \"8bbdf3c4-c287-4af3-be2c-6af952e8a9c2\") " Jul 16 21:55:15.228214 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:15.228104 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8bbdf3c4-c287-4af3-be2c-6af952e8a9c2-util\") pod \"8bbdf3c4-c287-4af3-be2c-6af952e8a9c2\" (UID: \"8bbdf3c4-c287-4af3-be2c-6af952e8a9c2\") " Jul 16 21:55:15.228503 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:15.228475 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bbdf3c4-c287-4af3-be2c-6af952e8a9c2-bundle" (OuterVolumeSpecName: "bundle") pod "8bbdf3c4-c287-4af3-be2c-6af952e8a9c2" (UID: "8bbdf3c4-c287-4af3-be2c-6af952e8a9c2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:55:15.230157 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:15.230131 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bbdf3c4-c287-4af3-be2c-6af952e8a9c2-kube-api-access-z854t" (OuterVolumeSpecName: "kube-api-access-z854t") pod "8bbdf3c4-c287-4af3-be2c-6af952e8a9c2" (UID: "8bbdf3c4-c287-4af3-be2c-6af952e8a9c2"). InnerVolumeSpecName "kube-api-access-z854t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:55:15.233300 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:15.233242 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bbdf3c4-c287-4af3-be2c-6af952e8a9c2-util" (OuterVolumeSpecName: "util") pod "8bbdf3c4-c287-4af3-be2c-6af952e8a9c2" (UID: "8bbdf3c4-c287-4af3-be2c-6af952e8a9c2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:55:15.329569 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:15.329539 2585 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8bbdf3c4-c287-4af3-be2c-6af952e8a9c2-bundle\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:55:15.329569 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:15.329564 2585 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8bbdf3c4-c287-4af3-be2c-6af952e8a9c2-util\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:55:15.329569 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:15.329575 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z854t\" (UniqueName: \"kubernetes.io/projected/8bbdf3c4-c287-4af3-be2c-6af952e8a9c2-kube-api-access-z854t\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:55:15.992998 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:15.992966 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596njrj" event={"ID":"8bbdf3c4-c287-4af3-be2c-6af952e8a9c2","Type":"ContainerDied","Data":"a78ef23ea7636fcbbce929a8212c379748d72a42671d6991f55d712c030a4d2c"} Jul 16 21:55:15.993384 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:15.993016 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a78ef23ea7636fcbbce929a8212c379748d72a42671d6991f55d712c030a4d2c" Jul 16 21:55:15.993384 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:15.992973 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596njrj" Jul 16 21:55:15.995079 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:15.995048 2585 generic.go:356] "Generic (PLEG): container finished" podID="bc100a65-eb66-4a34-9e07-7fd60fe7d3f5" containerID="74829a2ff241c7ab5debdbf567980c85351af3d97e702db9decb360d138a2b18" exitCode=0 Jul 16 21:55:15.995233 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:15.995082 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e089f4p" event={"ID":"bc100a65-eb66-4a34-9e07-7fd60fe7d3f5","Type":"ContainerDied","Data":"74829a2ff241c7ab5debdbf567980c85351af3d97e702db9decb360d138a2b18"} Jul 16 21:55:16.150049 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:16.150025 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739qtkh" Jul 16 21:55:16.152559 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:16.152539 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fvbjb" Jul 16 21:55:16.237313 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:16.237282 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7b6c3b0d-c66b-46ee-b013-b5bb9e619888-util\") pod \"7b6c3b0d-c66b-46ee-b013-b5bb9e619888\" (UID: \"7b6c3b0d-c66b-46ee-b013-b5bb9e619888\") " Jul 16 21:55:16.237465 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:16.237393 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1deb487d-8753-4dc6-a838-2c20e9841647-util\") pod \"1deb487d-8753-4dc6-a838-2c20e9841647\" (UID: \"1deb487d-8753-4dc6-a838-2c20e9841647\") " Jul 16 21:55:16.237465 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:16.237424 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrw5v\" (UniqueName: \"kubernetes.io/projected/1deb487d-8753-4dc6-a838-2c20e9841647-kube-api-access-lrw5v\") pod \"1deb487d-8753-4dc6-a838-2c20e9841647\" (UID: \"1deb487d-8753-4dc6-a838-2c20e9841647\") " Jul 16 21:55:16.237465 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:16.237453 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7b6c3b0d-c66b-46ee-b013-b5bb9e619888-bundle\") pod \"7b6c3b0d-c66b-46ee-b013-b5bb9e619888\" (UID: \"7b6c3b0d-c66b-46ee-b013-b5bb9e619888\") " Jul 16 21:55:16.237591 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:16.237486 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lczf\" (UniqueName: \"kubernetes.io/projected/7b6c3b0d-c66b-46ee-b013-b5bb9e619888-kube-api-access-5lczf\") pod \"7b6c3b0d-c66b-46ee-b013-b5bb9e619888\" (UID: \"7b6c3b0d-c66b-46ee-b013-b5bb9e619888\") " Jul 16 21:55:16.237591 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:16.237527 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1deb487d-8753-4dc6-a838-2c20e9841647-bundle\") pod \"1deb487d-8753-4dc6-a838-2c20e9841647\" (UID: \"1deb487d-8753-4dc6-a838-2c20e9841647\") " Jul 16 21:55:16.238205 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:16.238173 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1deb487d-8753-4dc6-a838-2c20e9841647-bundle" (OuterVolumeSpecName: "bundle") pod "1deb487d-8753-4dc6-a838-2c20e9841647" (UID: "1deb487d-8753-4dc6-a838-2c20e9841647"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:55:16.238360 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:16.238305 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b6c3b0d-c66b-46ee-b013-b5bb9e619888-bundle" (OuterVolumeSpecName: "bundle") pod "7b6c3b0d-c66b-46ee-b013-b5bb9e619888" (UID: "7b6c3b0d-c66b-46ee-b013-b5bb9e619888"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:55:16.239526 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:16.239505 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1deb487d-8753-4dc6-a838-2c20e9841647-kube-api-access-lrw5v" (OuterVolumeSpecName: "kube-api-access-lrw5v") pod "1deb487d-8753-4dc6-a838-2c20e9841647" (UID: "1deb487d-8753-4dc6-a838-2c20e9841647"). InnerVolumeSpecName "kube-api-access-lrw5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:55:16.239873 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:16.239857 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b6c3b0d-c66b-46ee-b013-b5bb9e619888-kube-api-access-5lczf" (OuterVolumeSpecName: "kube-api-access-5lczf") pod "7b6c3b0d-c66b-46ee-b013-b5bb9e619888" (UID: "7b6c3b0d-c66b-46ee-b013-b5bb9e619888"). InnerVolumeSpecName "kube-api-access-5lczf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:55:16.243334 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:16.243293 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1deb487d-8753-4dc6-a838-2c20e9841647-util" (OuterVolumeSpecName: "util") pod "1deb487d-8753-4dc6-a838-2c20e9841647" (UID: "1deb487d-8753-4dc6-a838-2c20e9841647"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:55:16.243602 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:16.243582 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b6c3b0d-c66b-46ee-b013-b5bb9e619888-util" (OuterVolumeSpecName: "util") pod "7b6c3b0d-c66b-46ee-b013-b5bb9e619888" (UID: "7b6c3b0d-c66b-46ee-b013-b5bb9e619888"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:55:16.338977 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:16.338889 2585 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1deb487d-8753-4dc6-a838-2c20e9841647-bundle\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:55:16.338977 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:16.338924 2585 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7b6c3b0d-c66b-46ee-b013-b5bb9e619888-util\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:55:16.338977 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:16.338933 2585 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1deb487d-8753-4dc6-a838-2c20e9841647-util\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:55:16.338977 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:16.338941 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrw5v\" (UniqueName: \"kubernetes.io/projected/1deb487d-8753-4dc6-a838-2c20e9841647-kube-api-access-lrw5v\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:55:16.338977 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:16.338951 2585 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7b6c3b0d-c66b-46ee-b013-b5bb9e619888-bundle\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:55:16.338977 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:16.338959 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lczf\" (UniqueName: \"kubernetes.io/projected/7b6c3b0d-c66b-46ee-b013-b5bb9e619888-kube-api-access-5lczf\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:55:17.007212 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:17.007180 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739qtkh" Jul 16 21:55:17.007689 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:17.007181 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739qtkh" event={"ID":"7b6c3b0d-c66b-46ee-b013-b5bb9e619888","Type":"ContainerDied","Data":"e64abe77efe8ff746d90ecfb9ebee081dab1dd2d77edb68cee81b7099e0bca0c"} Jul 16 21:55:17.007689 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:17.007318 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e64abe77efe8ff746d90ecfb9ebee081dab1dd2d77edb68cee81b7099e0bca0c" Jul 16 21:55:17.008997 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:17.008974 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fvbjb" event={"ID":"1deb487d-8753-4dc6-a838-2c20e9841647","Type":"ContainerDied","Data":"63cb49346e32669f8a7c3f822911d71328cb64b23d81045c5152ba18b0d863d8"} Jul 16 21:55:17.009132 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:17.009005 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fvbjb" Jul 16 21:55:17.009132 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:17.009012 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63cb49346e32669f8a7c3f822911d71328cb64b23d81045c5152ba18b0d863d8" Jul 16 21:55:17.134779 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:17.134757 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e089f4p" Jul 16 21:55:17.246594 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:17.246570 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjsfz\" (UniqueName: \"kubernetes.io/projected/bc100a65-eb66-4a34-9e07-7fd60fe7d3f5-kube-api-access-cjsfz\") pod \"bc100a65-eb66-4a34-9e07-7fd60fe7d3f5\" (UID: \"bc100a65-eb66-4a34-9e07-7fd60fe7d3f5\") " Jul 16 21:55:17.246741 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:17.246620 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc100a65-eb66-4a34-9e07-7fd60fe7d3f5-util\") pod \"bc100a65-eb66-4a34-9e07-7fd60fe7d3f5\" (UID: \"bc100a65-eb66-4a34-9e07-7fd60fe7d3f5\") " Jul 16 21:55:17.246741 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:17.246675 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc100a65-eb66-4a34-9e07-7fd60fe7d3f5-bundle\") pod \"bc100a65-eb66-4a34-9e07-7fd60fe7d3f5\" (UID: \"bc100a65-eb66-4a34-9e07-7fd60fe7d3f5\") " Jul 16 21:55:17.247191 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:17.247161 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc100a65-eb66-4a34-9e07-7fd60fe7d3f5-bundle" (OuterVolumeSpecName: "bundle") pod "bc100a65-eb66-4a34-9e07-7fd60fe7d3f5" (UID: "bc100a65-eb66-4a34-9e07-7fd60fe7d3f5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:55:17.248679 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:17.248640 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc100a65-eb66-4a34-9e07-7fd60fe7d3f5-kube-api-access-cjsfz" (OuterVolumeSpecName: "kube-api-access-cjsfz") pod "bc100a65-eb66-4a34-9e07-7fd60fe7d3f5" (UID: "bc100a65-eb66-4a34-9e07-7fd60fe7d3f5"). InnerVolumeSpecName "kube-api-access-cjsfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:55:17.252026 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:17.252003 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc100a65-eb66-4a34-9e07-7fd60fe7d3f5-util" (OuterVolumeSpecName: "util") pod "bc100a65-eb66-4a34-9e07-7fd60fe7d3f5" (UID: "bc100a65-eb66-4a34-9e07-7fd60fe7d3f5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:55:17.347965 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:17.347941 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjsfz\" (UniqueName: \"kubernetes.io/projected/bc100a65-eb66-4a34-9e07-7fd60fe7d3f5-kube-api-access-cjsfz\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:55:17.347965 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:17.347965 2585 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc100a65-eb66-4a34-9e07-7fd60fe7d3f5-util\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:55:17.348104 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:17.347975 2585 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc100a65-eb66-4a34-9e07-7fd60fe7d3f5-bundle\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:55:18.015736 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:18.015705 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e089f4p" event={"ID":"bc100a65-eb66-4a34-9e07-7fd60fe7d3f5","Type":"ContainerDied","Data":"dbc7832b45a8c4988b1888c1d3060d8fc97168b52e5665464b16dc13fe805933"} Jul 16 21:55:18.016166 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:18.015760 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbc7832b45a8c4988b1888c1d3060d8fc97168b52e5665464b16dc13fe805933" Jul 16 21:55:18.016166 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:18.015715 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e089f4p" Jul 16 21:55:19.733369 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:19.733334 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:55:20.349558 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:20.349528 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:55:20.823744 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:20.823718 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:55:21.024448 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:21.024416 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:55:22.145343 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:22.145315 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:55:24.775946 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:24.775908 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:55:25.245656 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:25.245622 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:55:25.530024 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:25.529920 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:55:27.160411 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:27.160377 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:55:29.005085 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:29.005053 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:55:30.856517 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:30.856443 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:55:31.787856 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:31.787816 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:55:33.104161 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:33.104132 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:55:33.638236 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:33.638191 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:55:33.895491 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:33.895413 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:55:35.580627 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:35.580596 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:55:35.754038 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:35.754001 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:55:35.941108 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:35.941022 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:55:36.114950 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:36.114922 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:55:39.656844 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:39.656814 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:55:40.067939 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:40.067905 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:55:40.563104 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:40.563071 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:55:40.991365 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:40.991288 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:55:41.468877 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:41.468845 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:55:47.698570 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:47.698535 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:55:48.296458 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:48.296423 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:55:50.278921 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:50.278885 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:55:50.615319 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:55:50.615290 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:56:27.236704 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:27.236616 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:56:27.737807 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:27.737771 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:56:29.720131 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:29.720098 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:56:29.805961 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:29.805927 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:56:29.907447 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:29.907411 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:56:30.203134 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.203099 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 21:56:30.203434 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.203421 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bbdf3c4-c287-4af3-be2c-6af952e8a9c2" containerName="util" Jul 16 21:56:30.203489 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.203437 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bbdf3c4-c287-4af3-be2c-6af952e8a9c2" containerName="util" Jul 16 21:56:30.203489 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.203448 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1deb487d-8753-4dc6-a838-2c20e9841647" containerName="util" Jul 16 21:56:30.203489 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.203454 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1deb487d-8753-4dc6-a838-2c20e9841647" containerName="util" Jul 16 21:56:30.203489 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.203461 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bbdf3c4-c287-4af3-be2c-6af952e8a9c2" containerName="pull" Jul 16 21:56:30.203489 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.203468 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bbdf3c4-c287-4af3-be2c-6af952e8a9c2" containerName="pull" Jul 16 21:56:30.203489 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.203477 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc100a65-eb66-4a34-9e07-7fd60fe7d3f5" containerName="pull" Jul 16 21:56:30.203489 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.203483 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc100a65-eb66-4a34-9e07-7fd60fe7d3f5" containerName="pull" Jul 16 21:56:30.203489 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.203489 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc100a65-eb66-4a34-9e07-7fd60fe7d3f5" containerName="extract" Jul 16 21:56:30.203750 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.203494 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc100a65-eb66-4a34-9e07-7fd60fe7d3f5" containerName="extract" Jul 16 21:56:30.203750 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.203504 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1deb487d-8753-4dc6-a838-2c20e9841647" containerName="pull" Jul 16 21:56:30.203750 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.203508 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1deb487d-8753-4dc6-a838-2c20e9841647" containerName="pull" Jul 16 21:56:30.203750 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.203514 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b6c3b0d-c66b-46ee-b013-b5bb9e619888" containerName="util" Jul 16 21:56:30.203750 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.203519 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b6c3b0d-c66b-46ee-b013-b5bb9e619888" containerName="util" Jul 16 21:56:30.203750 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.203527 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc100a65-eb66-4a34-9e07-7fd60fe7d3f5" containerName="util" Jul 16 21:56:30.203750 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.203533 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc100a65-eb66-4a34-9e07-7fd60fe7d3f5" containerName="util" Jul 16 21:56:30.203750 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.203540 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b6c3b0d-c66b-46ee-b013-b5bb9e619888" containerName="extract" Jul 16 21:56:30.203750 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.203545 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b6c3b0d-c66b-46ee-b013-b5bb9e619888" containerName="extract" Jul 16 21:56:30.203750 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.203550 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1deb487d-8753-4dc6-a838-2c20e9841647" containerName="extract" Jul 16 21:56:30.203750 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.203555 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1deb487d-8753-4dc6-a838-2c20e9841647" containerName="extract" Jul 16 21:56:30.203750 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.203561 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bbdf3c4-c287-4af3-be2c-6af952e8a9c2" containerName="extract" Jul 16 21:56:30.203750 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.203566 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bbdf3c4-c287-4af3-be2c-6af952e8a9c2" containerName="extract" Jul 16 21:56:30.203750 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.203572 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b6c3b0d-c66b-46ee-b013-b5bb9e619888" containerName="pull" Jul 16 21:56:30.203750 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.203577 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b6c3b0d-c66b-46ee-b013-b5bb9e619888" containerName="pull" Jul 16 21:56:30.203750 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.203625 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc100a65-eb66-4a34-9e07-7fd60fe7d3f5" containerName="extract" Jul 16 21:56:30.203750 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.203634 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bbdf3c4-c287-4af3-be2c-6af952e8a9c2" containerName="extract" Jul 16 21:56:30.203750 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.203643 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b6c3b0d-c66b-46ee-b013-b5bb9e619888" containerName="extract" Jul 16 21:56:30.203750 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.203649 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="1deb487d-8753-4dc6-a838-2c20e9841647" containerName="extract" Jul 16 21:56:30.206510 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.206489 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-cth87" Jul 16 21:56:30.214244 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.214204 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rbwj4\"" Jul 16 21:56:30.215261 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.215247 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 16 21:56:30.215319 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.215256 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 16 21:56:30.215319 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.215251 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 16 21:56:30.219242 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.219199 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 21:56:30.270924 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.270882 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 21:56:30.328910 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.328872 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d95gt\" (UniqueName: \"kubernetes.io/projected/33460968-f59d-4d7f-af09-731b58eaf8de-kube-api-access-d95gt\") pod \"limitador-limitador-54d6d988d5-cth87\" (UID: \"33460968-f59d-4d7f-af09-731b58eaf8de\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-cth87" Jul 16 21:56:30.329089 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.328952 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/33460968-f59d-4d7f-af09-731b58eaf8de-config-file\") pod \"limitador-limitador-54d6d988d5-cth87\" (UID: \"33460968-f59d-4d7f-af09-731b58eaf8de\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-cth87" Jul 16 21:56:30.429884 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.429854 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d95gt\" (UniqueName: \"kubernetes.io/projected/33460968-f59d-4d7f-af09-731b58eaf8de-kube-api-access-d95gt\") pod \"limitador-limitador-54d6d988d5-cth87\" (UID: \"33460968-f59d-4d7f-af09-731b58eaf8de\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-cth87" Jul 16 21:56:30.430021 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.429908 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/33460968-f59d-4d7f-af09-731b58eaf8de-config-file\") pod \"limitador-limitador-54d6d988d5-cth87\" (UID: \"33460968-f59d-4d7f-af09-731b58eaf8de\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-cth87" Jul 16 21:56:30.430483 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.430467 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/33460968-f59d-4d7f-af09-731b58eaf8de-config-file\") pod \"limitador-limitador-54d6d988d5-cth87\" (UID: \"33460968-f59d-4d7f-af09-731b58eaf8de\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-cth87" Jul 16 21:56:30.445897 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.445868 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d95gt\" (UniqueName: \"kubernetes.io/projected/33460968-f59d-4d7f-af09-731b58eaf8de-kube-api-access-d95gt\") pod \"limitador-limitador-54d6d988d5-cth87\" (UID: \"33460968-f59d-4d7f-af09-731b58eaf8de\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-cth87" Jul 16 21:56:30.516800 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.516746 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-cth87" Jul 16 21:56:30.662718 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.662688 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 21:56:30.665786 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.665753 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:56:30.666740 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:56:30.666719 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33460968_f59d_4d7f_af09_731b58eaf8de.slice/crio-a0a9e4fdb03bb25b33d86d726af6d1c008e885a15efa56ef0610823fe5ca728b WatchSource:0}: Error finding container a0a9e4fdb03bb25b33d86d726af6d1c008e885a15efa56ef0610823fe5ca728b: Status 404 returned error can't find the container with id a0a9e4fdb03bb25b33d86d726af6d1c008e885a15efa56ef0610823fe5ca728b Jul 16 21:56:30.767100 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.767031 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:56:30.995961 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:30.995927 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-lp5h5"] Jul 16 21:56:31.000255 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:31.000233 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-lp5h5" Jul 16 21:56:31.014976 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:31.014949 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-lp5h5"] Jul 16 21:56:31.035605 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:31.035533 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7mkz\" (UniqueName: \"kubernetes.io/projected/508ef640-4939-416c-8a30-1e1bbb6fa573-kube-api-access-r7mkz\") pod \"authorino-75d556bd77-lp5h5\" (UID: \"508ef640-4939-416c-8a30-1e1bbb6fa573\") " pod="kuadrant-system/authorino-75d556bd77-lp5h5" Jul 16 21:56:31.041808 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:31.041789 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wjtr7\"" Jul 16 21:56:31.136883 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:31.136480 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7mkz\" (UniqueName: \"kubernetes.io/projected/508ef640-4939-416c-8a30-1e1bbb6fa573-kube-api-access-r7mkz\") pod \"authorino-75d556bd77-lp5h5\" (UID: \"508ef640-4939-416c-8a30-1e1bbb6fa573\") " pod="kuadrant-system/authorino-75d556bd77-lp5h5" Jul 16 21:56:31.148205 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:31.148148 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7mkz\" (UniqueName: \"kubernetes.io/projected/508ef640-4939-416c-8a30-1e1bbb6fa573-kube-api-access-r7mkz\") pod \"authorino-75d556bd77-lp5h5\" (UID: \"508ef640-4939-416c-8a30-1e1bbb6fa573\") " pod="kuadrant-system/authorino-75d556bd77-lp5h5" Jul 16 21:56:31.309663 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:31.309630 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-lp5h5" Jul 16 21:56:31.382068 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:31.382033 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-cth87" event={"ID":"33460968-f59d-4d7f-af09-731b58eaf8de","Type":"ContainerStarted","Data":"a0a9e4fdb03bb25b33d86d726af6d1c008e885a15efa56ef0610823fe5ca728b"} Jul 16 21:56:31.467447 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:31.467413 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-lp5h5"] Jul 16 21:56:31.468586 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:56:31.468548 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod508ef640_4939_416c_8a30_1e1bbb6fa573.slice/crio-6c57074b6364fde13b849fca3039a8d1b1b5ca79b4f0131758e4505abd289c3b WatchSource:0}: Error finding container 6c57074b6364fde13b849fca3039a8d1b1b5ca79b4f0131758e4505abd289c3b: Status 404 returned error can't find the container with id 6c57074b6364fde13b849fca3039a8d1b1b5ca79b4f0131758e4505abd289c3b Jul 16 21:56:32.388803 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:32.388755 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-lp5h5" event={"ID":"508ef640-4939-416c-8a30-1e1bbb6fa573","Type":"ContainerStarted","Data":"6c57074b6364fde13b849fca3039a8d1b1b5ca79b4f0131758e4505abd289c3b"} Jul 16 21:56:33.398407 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:33.398371 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-cth87" event={"ID":"33460968-f59d-4d7f-af09-731b58eaf8de","Type":"ContainerStarted","Data":"ef106cfcb4c17a9b3ccadbf96c32351fd0200f343d03f5857eff03ca1b194e84"} Jul 16 21:56:33.398881 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:33.398593 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-cth87" Jul 16 21:56:33.425379 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:33.425326 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-cth87" podStartSLOduration=0.869679554 podStartE2EDuration="3.425307606s" totalImagesPullingTime="2.555628052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:56:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:56:33.424308742 +0000 UTC m=+542.046751695" watchObservedRunningTime="2026-07-16 21:56:33.425307606 +0000 UTC m=+542.047750560" Jul 16 21:56:35.411366 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:35.411327 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-lp5h5" event={"ID":"508ef640-4939-416c-8a30-1e1bbb6fa573","Type":"ContainerStarted","Data":"e9b8008722b98872f9ac3651fd1812a83ce5ae79b8ae121c0f84be1a95219368"} Jul 16 21:56:35.429410 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:35.429364 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-lp5h5" podStartSLOduration=1.636997917 podStartE2EDuration="5.429350215s" totalImagesPullingTime="3.792352298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:56:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:56:35.428563104 +0000 UTC m=+544.051006058" watchObservedRunningTime="2026-07-16 21:56:35.429350215 +0000 UTC m=+544.051793205" Jul 16 21:56:37.422652 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:37.422624 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:56:44.405438 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:44.405403 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-cth87" Jul 16 21:56:45.934053 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:45.934022 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:56:46.128547 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:46.128507 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:56:46.481327 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:46.481299 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:56:49.412139 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:49.412104 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:56:58.105389 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:58.105357 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:56:58.521080 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:58.520997 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:56:58.828989 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:58.828950 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7576fd57d8-qf265"] Jul 16 21:56:58.832334 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:58.832317 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7576fd57d8-qf265" Jul 16 21:56:58.835440 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:58.835420 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 16 21:56:58.842763 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:58.842740 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7576fd57d8-qf265"] Jul 16 21:56:58.982018 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:58.981988 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ea599498-2838-4ba4-b630-3bf6f13fb007-tls-cert\") pod \"authorino-7576fd57d8-qf265\" (UID: \"ea599498-2838-4ba4-b630-3bf6f13fb007\") " pod="kuadrant-system/authorino-7576fd57d8-qf265" Jul 16 21:56:58.982196 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:58.982033 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4t88\" (UniqueName: \"kubernetes.io/projected/ea599498-2838-4ba4-b630-3bf6f13fb007-kube-api-access-r4t88\") pod \"authorino-7576fd57d8-qf265\" (UID: \"ea599498-2838-4ba4-b630-3bf6f13fb007\") " pod="kuadrant-system/authorino-7576fd57d8-qf265" Jul 16 21:56:59.083043 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:59.082960 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4t88\" (UniqueName: \"kubernetes.io/projected/ea599498-2838-4ba4-b630-3bf6f13fb007-kube-api-access-r4t88\") pod \"authorino-7576fd57d8-qf265\" (UID: \"ea599498-2838-4ba4-b630-3bf6f13fb007\") " pod="kuadrant-system/authorino-7576fd57d8-qf265" Jul 16 21:56:59.083195 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:59.083071 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ea599498-2838-4ba4-b630-3bf6f13fb007-tls-cert\") pod \"authorino-7576fd57d8-qf265\" (UID: \"ea599498-2838-4ba4-b630-3bf6f13fb007\") " pod="kuadrant-system/authorino-7576fd57d8-qf265" Jul 16 21:56:59.085629 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:59.085597 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ea599498-2838-4ba4-b630-3bf6f13fb007-tls-cert\") pod \"authorino-7576fd57d8-qf265\" (UID: \"ea599498-2838-4ba4-b630-3bf6f13fb007\") " pod="kuadrant-system/authorino-7576fd57d8-qf265" Jul 16 21:56:59.091749 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:59.091722 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4t88\" (UniqueName: \"kubernetes.io/projected/ea599498-2838-4ba4-b630-3bf6f13fb007-kube-api-access-r4t88\") pod \"authorino-7576fd57d8-qf265\" (UID: \"ea599498-2838-4ba4-b630-3bf6f13fb007\") " pod="kuadrant-system/authorino-7576fd57d8-qf265" Jul 16 21:56:59.142004 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:59.141978 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7576fd57d8-qf265" Jul 16 21:56:59.284660 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:59.284357 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7576fd57d8-qf265"] Jul 16 21:56:59.285839 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:56:59.285815 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea599498_2838_4ba4_b630_3bf6f13fb007.slice/crio-c8478317cb8541cd2fa1cb8aebfbbcda6d193051645ea7d5899b798bc8b2683a WatchSource:0}: Error finding container c8478317cb8541cd2fa1cb8aebfbbcda6d193051645ea7d5899b798bc8b2683a: Status 404 returned error can't find the container with id c8478317cb8541cd2fa1cb8aebfbbcda6d193051645ea7d5899b798bc8b2683a Jul 16 21:56:59.527545 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:56:59.527459 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7576fd57d8-qf265" event={"ID":"ea599498-2838-4ba4-b630-3bf6f13fb007","Type":"ContainerStarted","Data":"c8478317cb8541cd2fa1cb8aebfbbcda6d193051645ea7d5899b798bc8b2683a"} Jul 16 21:57:00.534935 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:57:00.534896 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7576fd57d8-qf265" event={"ID":"ea599498-2838-4ba4-b630-3bf6f13fb007","Type":"ContainerStarted","Data":"99e12be7ea2c53ee2a8ad716ab3ca69db9434e7e20f6287970a4483eb79dbeb2"} Jul 16 21:57:00.591778 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:57:00.591722 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7576fd57d8-qf265" podStartSLOduration=2.243460204 podStartE2EDuration="2.591703588s" totalImagesPullingTime="348.243384ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:56:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:57:00.557527416 +0000 UTC m=+569.179970369" watchObservedRunningTime="2026-07-16 21:57:00.591703588 +0000 UTC m=+569.214146541" Jul 16 21:57:00.592149 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:57:00.592134 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-lp5h5"] Jul 16 21:57:00.592329 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:57:00.592311 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-lp5h5" podUID="508ef640-4939-416c-8a30-1e1bbb6fa573" containerName="authorino" containerID="cri-o://e9b8008722b98872f9ac3651fd1812a83ce5ae79b8ae121c0f84be1a95219368" gracePeriod=30 Jul 16 21:57:00.840903 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:57:00.840880 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-lp5h5" Jul 16 21:57:01.002066 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:57:01.002030 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7mkz\" (UniqueName: \"kubernetes.io/projected/508ef640-4939-416c-8a30-1e1bbb6fa573-kube-api-access-r7mkz\") pod \"508ef640-4939-416c-8a30-1e1bbb6fa573\" (UID: \"508ef640-4939-416c-8a30-1e1bbb6fa573\") " Jul 16 21:57:01.004039 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:57:01.004009 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/508ef640-4939-416c-8a30-1e1bbb6fa573-kube-api-access-r7mkz" (OuterVolumeSpecName: "kube-api-access-r7mkz") pod "508ef640-4939-416c-8a30-1e1bbb6fa573" (UID: "508ef640-4939-416c-8a30-1e1bbb6fa573"). InnerVolumeSpecName "kube-api-access-r7mkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:57:01.102893 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:57:01.102817 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7mkz\" (UniqueName: \"kubernetes.io/projected/508ef640-4939-416c-8a30-1e1bbb6fa573-kube-api-access-r7mkz\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 21:57:01.541473 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:57:01.541382 2585 generic.go:356] "Generic (PLEG): container finished" podID="508ef640-4939-416c-8a30-1e1bbb6fa573" containerID="e9b8008722b98872f9ac3651fd1812a83ce5ae79b8ae121c0f84be1a95219368" exitCode=0 Jul 16 21:57:01.541473 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:57:01.541422 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-lp5h5" Jul 16 21:57:01.541928 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:57:01.541467 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-lp5h5" event={"ID":"508ef640-4939-416c-8a30-1e1bbb6fa573","Type":"ContainerDied","Data":"e9b8008722b98872f9ac3651fd1812a83ce5ae79b8ae121c0f84be1a95219368"} Jul 16 21:57:01.541928 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:57:01.541512 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-lp5h5" event={"ID":"508ef640-4939-416c-8a30-1e1bbb6fa573","Type":"ContainerDied","Data":"6c57074b6364fde13b849fca3039a8d1b1b5ca79b4f0131758e4505abd289c3b"} Jul 16 21:57:01.541928 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:57:01.541533 2585 scope.go:117] "RemoveContainer" containerID="e9b8008722b98872f9ac3651fd1812a83ce5ae79b8ae121c0f84be1a95219368" Jul 16 21:57:01.550657 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:57:01.550638 2585 scope.go:117] "RemoveContainer" containerID="e9b8008722b98872f9ac3651fd1812a83ce5ae79b8ae121c0f84be1a95219368" Jul 16 21:57:01.550916 ip-10-0-128-255 kubenswrapper[2585]: E0716 21:57:01.550893 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9b8008722b98872f9ac3651fd1812a83ce5ae79b8ae121c0f84be1a95219368\": container with ID starting with e9b8008722b98872f9ac3651fd1812a83ce5ae79b8ae121c0f84be1a95219368 not found: ID does not exist" containerID="e9b8008722b98872f9ac3651fd1812a83ce5ae79b8ae121c0f84be1a95219368" Jul 16 21:57:01.550978 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:57:01.550928 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9b8008722b98872f9ac3651fd1812a83ce5ae79b8ae121c0f84be1a95219368"} err="failed to get container status \"e9b8008722b98872f9ac3651fd1812a83ce5ae79b8ae121c0f84be1a95219368\": rpc error: code = NotFound desc = could not find container \"e9b8008722b98872f9ac3651fd1812a83ce5ae79b8ae121c0f84be1a95219368\": container with ID starting with e9b8008722b98872f9ac3651fd1812a83ce5ae79b8ae121c0f84be1a95219368 not found: ID does not exist" Jul 16 21:57:01.563657 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:57:01.563634 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-lp5h5"] Jul 16 21:57:01.568931 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:57:01.568909 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-lp5h5"] Jul 16 21:57:01.939849 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:57:01.939814 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="508ef640-4939-416c-8a30-1e1bbb6fa573" path="/var/lib/kubelet/pods/508ef640-4939-416c-8a30-1e1bbb6fa573/volumes" Jul 16 21:57:05.402087 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:57:05.402054 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:57:05.518134 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:57:05.518089 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:57:05.594700 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:57:05.594661 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:57:05.665066 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:57:05.664986 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:57:09.026609 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:57:09.026572 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:57:09.152729 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:57:09.152701 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:57:40.791063 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:57:40.791022 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:57:42.377164 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:57:42.377125 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:57:42.528826 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:57:42.528790 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:57:42.910654 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:57:42.910625 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 21:57:51.088063 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:57:51.088029 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:57:51.190457 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:57:51.190420 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:58:02.705240 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:58:02.705156 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:58:03.095592 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:58:03.095554 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 21:58:17.147296 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:58:17.147260 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:58:17.812121 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:58:17.812073 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 21:58:29.442449 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:58:29.442415 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:58:29.497691 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:58:29.497654 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:58:35.915346 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:58:35.915306 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:58:37.097143 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:58:37.097106 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 21:58:49.217370 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:58:49.217320 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:58:50.905886 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:58:50.905852 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 21:58:57.657579 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:58:57.657543 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:58:59.697891 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:58:59.697855 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 21:59:00.582144 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:59:00.582105 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb"] Jul 16 21:59:00.582703 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:59:00.582680 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="508ef640-4939-416c-8a30-1e1bbb6fa573" containerName="authorino" Jul 16 21:59:00.582786 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:59:00.582707 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="508ef640-4939-416c-8a30-1e1bbb6fa573" containerName="authorino" Jul 16 21:59:00.582837 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:59:00.582798 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="508ef640-4939-416c-8a30-1e1bbb6fa573" containerName="authorino" Jul 16 21:59:00.586084 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:59:00.586061 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb" Jul 16 21:59:00.588700 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:59:00.588674 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-5gqbs\"" Jul 16 21:59:00.588820 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:59:00.588757 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 16 21:59:00.589698 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:59:00.589658 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 16 21:59:00.589811 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:59:00.589700 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 16 21:59:00.596638 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:59:00.596577 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb"] Jul 16 21:59:00.611301 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:59:00.611270 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7053ba1-14fe-4849-ad04-509946c1e37b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb\" (UID: \"b7053ba1-14fe-4849-ad04-509946c1e37b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb" Jul 16 21:59:00.611421 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:59:00.611318 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7053ba1-14fe-4849-ad04-509946c1e37b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb\" (UID: \"b7053ba1-14fe-4849-ad04-509946c1e37b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb" Jul 16 21:59:00.611421 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:59:00.611357 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghj46\" (UniqueName: \"kubernetes.io/projected/b7053ba1-14fe-4849-ad04-509946c1e37b-kube-api-access-ghj46\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb\" (UID: \"b7053ba1-14fe-4849-ad04-509946c1e37b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb" Jul 16 21:59:00.611421 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:59:00.611407 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7053ba1-14fe-4849-ad04-509946c1e37b-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb\" (UID: \"b7053ba1-14fe-4849-ad04-509946c1e37b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb" Jul 16 21:59:00.611573 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:59:00.611475 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7053ba1-14fe-4849-ad04-509946c1e37b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb\" (UID: \"b7053ba1-14fe-4849-ad04-509946c1e37b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb" Jul 16 21:59:00.611573 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:59:00.611540 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7053ba1-14fe-4849-ad04-509946c1e37b-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb\" (UID: \"b7053ba1-14fe-4849-ad04-509946c1e37b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb" Jul 16 21:59:00.612012 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:59:00.611992 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:59:00.712723 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:59:00.712692 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7053ba1-14fe-4849-ad04-509946c1e37b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb\" (UID: \"b7053ba1-14fe-4849-ad04-509946c1e37b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb" Jul 16 21:59:00.713171 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:59:00.712729 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7053ba1-14fe-4849-ad04-509946c1e37b-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb\" (UID: \"b7053ba1-14fe-4849-ad04-509946c1e37b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb" Jul 16 21:59:00.713171 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:59:00.712759 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7053ba1-14fe-4849-ad04-509946c1e37b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb\" (UID: \"b7053ba1-14fe-4849-ad04-509946c1e37b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb" Jul 16 21:59:00.713171 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:59:00.712892 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7053ba1-14fe-4849-ad04-509946c1e37b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb\" (UID: \"b7053ba1-14fe-4849-ad04-509946c1e37b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb" Jul 16 21:59:00.713171 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:59:00.712949 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghj46\" (UniqueName: \"kubernetes.io/projected/b7053ba1-14fe-4849-ad04-509946c1e37b-kube-api-access-ghj46\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb\" (UID: \"b7053ba1-14fe-4849-ad04-509946c1e37b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb" Jul 16 21:59:00.713171 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:59:00.713016 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7053ba1-14fe-4849-ad04-509946c1e37b-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb\" (UID: \"b7053ba1-14fe-4849-ad04-509946c1e37b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb" Jul 16 21:59:00.713171 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:59:00.713111 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7053ba1-14fe-4849-ad04-509946c1e37b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb\" (UID: \"b7053ba1-14fe-4849-ad04-509946c1e37b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb" Jul 16 21:59:00.713171 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:59:00.713144 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7053ba1-14fe-4849-ad04-509946c1e37b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb\" (UID: \"b7053ba1-14fe-4849-ad04-509946c1e37b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb" Jul 16 21:59:00.713518 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:59:00.713360 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7053ba1-14fe-4849-ad04-509946c1e37b-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb\" (UID: \"b7053ba1-14fe-4849-ad04-509946c1e37b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb" Jul 16 21:59:00.715066 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:59:00.715045 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7053ba1-14fe-4849-ad04-509946c1e37b-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb\" (UID: \"b7053ba1-14fe-4849-ad04-509946c1e37b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb" Jul 16 21:59:00.715399 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:59:00.715378 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7053ba1-14fe-4849-ad04-509946c1e37b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb\" (UID: \"b7053ba1-14fe-4849-ad04-509946c1e37b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb" Jul 16 21:59:00.725614 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:59:00.725592 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghj46\" (UniqueName: \"kubernetes.io/projected/b7053ba1-14fe-4849-ad04-509946c1e37b-kube-api-access-ghj46\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb\" (UID: \"b7053ba1-14fe-4849-ad04-509946c1e37b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb" Jul 16 21:59:00.898176 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:59:00.898091 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb" Jul 16 21:59:01.037964 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:59:01.037937 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb"] Jul 16 21:59:01.039280 ip-10-0-128-255 kubenswrapper[2585]: W0716 21:59:01.039252 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7053ba1_14fe_4849_ad04_509946c1e37b.slice/crio-0f3d1620e77029b2aae00fe68bc30363c56bc614186a40fa533016dbeb228b8c WatchSource:0}: Error finding container 0f3d1620e77029b2aae00fe68bc30363c56bc614186a40fa533016dbeb228b8c: Status 404 returned error can't find the container with id 0f3d1620e77029b2aae00fe68bc30363c56bc614186a40fa533016dbeb228b8c Jul 16 21:59:01.041154 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:59:01.041114 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 21:59:01.160008 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:59:01.159914 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb" event={"ID":"b7053ba1-14fe-4849-ad04-509946c1e37b","Type":"ContainerStarted","Data":"0f3d1620e77029b2aae00fe68bc30363c56bc614186a40fa533016dbeb228b8c"} Jul 16 21:59:02.166540 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:59:02.166507 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb" event={"ID":"b7053ba1-14fe-4849-ad04-509946c1e37b","Type":"ContainerStarted","Data":"4a20e23dd6957b25e6b1bd64acafaa2bb72092a1f2a4792c1ef4fb249c919c02"} Jul 16 21:59:02.187359 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:59:02.187298 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb" podStartSLOduration=1.126987816 podStartE2EDuration="2.187282088s" totalImagesPullingTime="1.060294272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:59:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:59:02.184262584 +0000 UTC m=+690.806705536" watchObservedRunningTime="2026-07-16 21:59:02.187282088 +0000 UTC m=+690.809725037" Jul 16 21:59:07.895899 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:59:07.895868 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 21:59:08.593200 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:59:08.593168 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 21:59:10.898811 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:59:10.898778 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb" Jul 16 21:59:10.899194 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:59:10.899042 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb" Jul 16 21:59:10.911346 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:59:10.911322 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb" Jul 16 21:59:11.225246 ip-10-0-128-255 kubenswrapper[2585]: I0716 21:59:11.225147 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb" Jul 16 22:00:17.236377 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:00:17.236341 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-84df8b88d9-r55wz"] Jul 16 22:00:17.239782 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:00:17.239759 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84df8b88d9-r55wz" Jul 16 22:00:17.247661 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:00:17.247635 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-84df8b88d9-r55wz"] Jul 16 22:00:17.346789 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:00:17.346758 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6lrm\" (UniqueName: \"kubernetes.io/projected/3480c87a-2733-42df-83f1-8eaaa7287e84-kube-api-access-g6lrm\") pod \"authorino-84df8b88d9-r55wz\" (UID: \"3480c87a-2733-42df-83f1-8eaaa7287e84\") " pod="kuadrant-system/authorino-84df8b88d9-r55wz" Jul 16 22:00:17.346948 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:00:17.346806 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3480c87a-2733-42df-83f1-8eaaa7287e84-tls-cert\") pod \"authorino-84df8b88d9-r55wz\" (UID: \"3480c87a-2733-42df-83f1-8eaaa7287e84\") " pod="kuadrant-system/authorino-84df8b88d9-r55wz" Jul 16 22:00:17.448121 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:00:17.448087 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6lrm\" (UniqueName: \"kubernetes.io/projected/3480c87a-2733-42df-83f1-8eaaa7287e84-kube-api-access-g6lrm\") pod \"authorino-84df8b88d9-r55wz\" (UID: \"3480c87a-2733-42df-83f1-8eaaa7287e84\") " pod="kuadrant-system/authorino-84df8b88d9-r55wz" Jul 16 22:00:17.448314 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:00:17.448139 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3480c87a-2733-42df-83f1-8eaaa7287e84-tls-cert\") pod \"authorino-84df8b88d9-r55wz\" (UID: \"3480c87a-2733-42df-83f1-8eaaa7287e84\") " pod="kuadrant-system/authorino-84df8b88d9-r55wz" Jul 16 22:00:17.450559 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:00:17.450529 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3480c87a-2733-42df-83f1-8eaaa7287e84-tls-cert\") pod \"authorino-84df8b88d9-r55wz\" (UID: \"3480c87a-2733-42df-83f1-8eaaa7287e84\") " pod="kuadrant-system/authorino-84df8b88d9-r55wz" Jul 16 22:00:17.456404 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:00:17.456378 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6lrm\" (UniqueName: \"kubernetes.io/projected/3480c87a-2733-42df-83f1-8eaaa7287e84-kube-api-access-g6lrm\") pod \"authorino-84df8b88d9-r55wz\" (UID: \"3480c87a-2733-42df-83f1-8eaaa7287e84\") " pod="kuadrant-system/authorino-84df8b88d9-r55wz" Jul 16 22:00:17.549546 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:00:17.549521 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84df8b88d9-r55wz" Jul 16 22:00:17.560915 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:00:17.560883 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:00:17.687702 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:00:17.687674 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-84df8b88d9-r55wz"] Jul 16 22:00:17.689229 ip-10-0-128-255 kubenswrapper[2585]: W0716 22:00:17.689190 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3480c87a_2733_42df_83f1_8eaaa7287e84.slice/crio-bf2be68719d5a8a3a7c4067f8c312c59cec779a46c157f8b95ab7cd1f2422e14 WatchSource:0}: Error finding container bf2be68719d5a8a3a7c4067f8c312c59cec779a46c157f8b95ab7cd1f2422e14: Status 404 returned error can't find the container with id bf2be68719d5a8a3a7c4067f8c312c59cec779a46c157f8b95ab7cd1f2422e14 Jul 16 22:00:17.996585 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:00:17.996500 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:00:18.540405 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:00:18.540367 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84df8b88d9-r55wz" event={"ID":"3480c87a-2733-42df-83f1-8eaaa7287e84","Type":"ContainerStarted","Data":"bf2be68719d5a8a3a7c4067f8c312c59cec779a46c157f8b95ab7cd1f2422e14"} Jul 16 22:00:18.654983 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:00:18.654914 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:00:18.751849 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:00:18.751819 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:00:19.549841 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:00:19.549786 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84df8b88d9-r55wz" event={"ID":"3480c87a-2733-42df-83f1-8eaaa7287e84","Type":"ContainerStarted","Data":"176adff7a29a2a93aa58baeb65ccd81d840ddb628c311c5a9f9280a1d2b8d31e"} Jul 16 22:00:19.568230 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:00:19.568119 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-84df8b88d9-r55wz" podStartSLOduration=0.956424595 podStartE2EDuration="2.568097379s" totalImagesPullingTime="1.611672784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:00:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:00:19.566828724 +0000 UTC m=+768.189271680" watchObservedRunningTime="2026-07-16 22:00:19.568097379 +0000 UTC m=+768.190540334" Jul 16 22:00:19.600296 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:00:19.600260 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7576fd57d8-qf265"] Jul 16 22:00:19.600497 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:00:19.600471 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7576fd57d8-qf265" podUID="ea599498-2838-4ba4-b630-3bf6f13fb007" containerName="authorino" containerID="cri-o://99e12be7ea2c53ee2a8ad716ab3ca69db9434e7e20f6287970a4483eb79dbeb2" gracePeriod=30 Jul 16 22:00:19.855115 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:00:19.855092 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7576fd57d8-qf265" Jul 16 22:00:19.972864 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:00:19.972834 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4t88\" (UniqueName: \"kubernetes.io/projected/ea599498-2838-4ba4-b630-3bf6f13fb007-kube-api-access-r4t88\") pod \"ea599498-2838-4ba4-b630-3bf6f13fb007\" (UID: \"ea599498-2838-4ba4-b630-3bf6f13fb007\") " Jul 16 22:00:19.973005 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:00:19.972873 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ea599498-2838-4ba4-b630-3bf6f13fb007-tls-cert\") pod \"ea599498-2838-4ba4-b630-3bf6f13fb007\" (UID: \"ea599498-2838-4ba4-b630-3bf6f13fb007\") " Jul 16 22:00:19.975023 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:00:19.974989 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea599498-2838-4ba4-b630-3bf6f13fb007-kube-api-access-r4t88" (OuterVolumeSpecName: "kube-api-access-r4t88") pod "ea599498-2838-4ba4-b630-3bf6f13fb007" (UID: "ea599498-2838-4ba4-b630-3bf6f13fb007"). InnerVolumeSpecName "kube-api-access-r4t88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 22:00:19.982903 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:00:19.982871 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea599498-2838-4ba4-b630-3bf6f13fb007-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "ea599498-2838-4ba4-b630-3bf6f13fb007" (UID: "ea599498-2838-4ba4-b630-3bf6f13fb007"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 22:00:20.074234 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:00:20.074188 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4t88\" (UniqueName: \"kubernetes.io/projected/ea599498-2838-4ba4-b630-3bf6f13fb007-kube-api-access-r4t88\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:00:20.074234 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:00:20.074230 2585 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ea599498-2838-4ba4-b630-3bf6f13fb007-tls-cert\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:00:20.556330 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:00:20.556296 2585 generic.go:356] "Generic (PLEG): container finished" podID="ea599498-2838-4ba4-b630-3bf6f13fb007" containerID="99e12be7ea2c53ee2a8ad716ab3ca69db9434e7e20f6287970a4483eb79dbeb2" exitCode=0 Jul 16 22:00:20.556790 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:00:20.556340 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7576fd57d8-qf265" Jul 16 22:00:20.556790 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:00:20.556392 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7576fd57d8-qf265" event={"ID":"ea599498-2838-4ba4-b630-3bf6f13fb007","Type":"ContainerDied","Data":"99e12be7ea2c53ee2a8ad716ab3ca69db9434e7e20f6287970a4483eb79dbeb2"} Jul 16 22:00:20.556790 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:00:20.556438 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7576fd57d8-qf265" event={"ID":"ea599498-2838-4ba4-b630-3bf6f13fb007","Type":"ContainerDied","Data":"c8478317cb8541cd2fa1cb8aebfbbcda6d193051645ea7d5899b798bc8b2683a"} Jul 16 22:00:20.556790 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:00:20.556459 2585 scope.go:117] "RemoveContainer" containerID="99e12be7ea2c53ee2a8ad716ab3ca69db9434e7e20f6287970a4483eb79dbeb2" Jul 16 22:00:20.568464 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:00:20.568446 2585 scope.go:117] "RemoveContainer" containerID="99e12be7ea2c53ee2a8ad716ab3ca69db9434e7e20f6287970a4483eb79dbeb2" Jul 16 22:00:20.568735 ip-10-0-128-255 kubenswrapper[2585]: E0716 22:00:20.568713 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99e12be7ea2c53ee2a8ad716ab3ca69db9434e7e20f6287970a4483eb79dbeb2\": container with ID starting with 99e12be7ea2c53ee2a8ad716ab3ca69db9434e7e20f6287970a4483eb79dbeb2 not found: ID does not exist" containerID="99e12be7ea2c53ee2a8ad716ab3ca69db9434e7e20f6287970a4483eb79dbeb2" Jul 16 22:00:20.568806 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:00:20.568742 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99e12be7ea2c53ee2a8ad716ab3ca69db9434e7e20f6287970a4483eb79dbeb2"} err="failed to get container status \"99e12be7ea2c53ee2a8ad716ab3ca69db9434e7e20f6287970a4483eb79dbeb2\": rpc error: code = NotFound desc = could not find container \"99e12be7ea2c53ee2a8ad716ab3ca69db9434e7e20f6287970a4483eb79dbeb2\": container with ID starting with 99e12be7ea2c53ee2a8ad716ab3ca69db9434e7e20f6287970a4483eb79dbeb2 not found: ID does not exist" Jul 16 22:00:20.581600 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:00:20.581576 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7576fd57d8-qf265"] Jul 16 22:00:20.584471 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:00:20.584450 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7576fd57d8-qf265"] Jul 16 22:00:21.934975 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:00:21.934946 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea599498-2838-4ba4-b630-3bf6f13fb007" path="/var/lib/kubelet/pods/ea599498-2838-4ba4-b630-3bf6f13fb007/volumes" Jul 16 22:00:46.499541 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:00:46.499502 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:00:52.267978 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:00:52.267943 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:00:55.788151 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:00:55.788072 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:01:06.412387 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:01:06.412352 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:01:24.099791 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:01:24.099743 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:01:24.159794 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:01:24.159772 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:01:49.449549 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:01:49.449509 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:02:11.324197 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:02:11.324161 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:02:22.000179 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:02:22.000137 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:02:23.266410 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:02:23.266364 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:02:43.094445 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:02:43.094404 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:02:47.303638 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:02:47.303602 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:02:56.615082 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:02:56.615042 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:02:59.102004 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:02:59.101973 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:02:59.193888 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:02:59.193859 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:02:59.828604 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:02:59.828569 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:03:06.855792 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:03:06.855754 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:03:07.037426 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:03:07.037394 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:03:07.291582 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:03:07.291493 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:03:13.855411 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:03:13.855374 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:03:13.955763 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:03:13.955731 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:03:30.817288 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:03:30.817238 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:03:40.200583 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:03:40.200543 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:03:49.137920 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:03:49.137882 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:04:00.803273 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:04:00.803187 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:04:13.302445 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:04:13.302408 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:04:22.638343 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:04:22.638305 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:04:34.501043 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:04:34.501008 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:04:44.497879 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:04:44.497841 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:05:09.801237 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:05:09.801180 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:05:19.405955 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:05:19.405918 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:05:28.560720 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:05:28.560639 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:05:43.900359 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:05:43.900319 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:05:53.791898 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:05:53.791862 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:06:09.294617 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:06:09.294580 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:06:18.597795 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:06:18.597760 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:06:29.717017 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:06:29.716974 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:06:38.601015 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:06:38.600979 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:06:47.797207 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:06:47.797169 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:08:38.004967 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:08:38.004929 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:08:49.003253 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:08:49.001478 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:08:58.203231 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:08:58.203191 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:09:09.294710 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:09:09.294676 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:09:19.795540 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:09:19.795503 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:09:31.722231 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:09:31.722187 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:09:40.609699 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:09:40.609657 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:09:51.696275 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:09:51.696238 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:10:00.913593 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:10:00.913504 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:10:12.103113 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:10:12.103081 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:10:24.610939 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:10:24.610907 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:10:38.666092 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:10:38.666055 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:10:40.054184 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:10:40.054133 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:10:45.802367 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:10:45.802332 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:10:54.799038 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:10:54.799001 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:11:04.605874 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:11:04.605839 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:11:16.991983 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:11:16.991947 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:11:25.106421 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:11:25.106325 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:11:33.600379 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:11:33.600342 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:11:35.413331 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:11:35.413293 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:11:35.511983 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:11:35.511955 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:11:43.498267 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:11:43.498230 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:11:52.910491 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:11:52.910456 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:12:03.404756 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:12:03.404717 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:12:12.304946 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:12:12.304912 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:12:22.492959 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:12:22.492923 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:12:23.378351 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:12:23.378304 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:12:35.794825 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:12:35.794789 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:12:44.795536 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:12:44.795503 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:12:54.506349 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:12:54.506270 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:13:04.405148 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:13:04.405110 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:13:17.399127 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:13:17.399086 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:13:26.107974 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:13:26.107935 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:13:40.812454 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:13:40.812419 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:13:52.393487 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:13:52.393455 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:14:07.359445 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:14:07.359402 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:14:07.739069 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:14:07.738990 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:14:16.890649 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:14:16.890612 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:14:20.144602 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:14:20.140127 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:14:25.157853 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:14:25.157778 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:14:30.410785 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:14:30.410750 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:14:33.339365 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:14:33.339312 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:14:33.449740 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:14:33.449708 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:14:33.775248 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:14:33.775143 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:14:34.408580 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:14:34.408548 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:14:39.642985 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:14:39.642934 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:14:39.714162 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:14:39.714131 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:14:39.874584 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:14:39.874553 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:14:39.972160 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:14:39.972080 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:14:41.118276 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:14:41.118248 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:14:41.220014 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:14:41.219986 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:14:41.397515 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:14:41.397433 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:14:41.535294 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:14:41.535268 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:14:42.018839 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:14:42.018810 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:14:42.142544 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:14:42.142510 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:14:43.757112 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:14:43.757054 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:14:45.022022 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:14:45.021981 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:17:05.726508 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:17:05.726474 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:17:44.573949 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:17:44.573916 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:17:44.675276 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:17:44.675231 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:17:46.146979 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:17:46.146947 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:17:46.258152 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:17:46.258117 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:17:48.797831 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:17:48.797798 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:17:49.382252 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:17:49.382199 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:17:51.907599 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:17:51.907564 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:17:52.249773 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:17:52.249693 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:17:57.705147 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:17:57.705121 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:17:58.199151 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:17:58.199090 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:17:58.300460 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:17:58.300432 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:17:58.467633 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:17:58.467553 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:17:58.575291 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:17:58.575252 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:17:59.077474 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:17:59.077445 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:17:59.177993 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:17:59.177963 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:18:00.032691 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:18:00.032658 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:18:00.178559 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:18:00.178531 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:18:00.316522 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:18:00.316495 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:18:00.660511 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:18:00.660424 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:18:00.763546 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:18:00.763513 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:18:02.407025 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:18:02.406990 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:18:02.584892 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:18:02.584862 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:18:44.131458 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:18:44.131422 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:18:44.268151 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:18:44.268122 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:19:04.672843 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:19:04.672759 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:19:04.782620 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:19:04.782587 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:19:07.423079 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:19:07.423044 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:19:10.606818 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:19:10.606781 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:19:11.437078 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:19:11.437030 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:19:12.140471 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:19:12.140441 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:19:18.351846 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:19:18.351813 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:19:18.458590 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:19:18.458559 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:19:18.709408 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:19:18.709374 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:19:18.813705 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:19:18.813677 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:19:20.304078 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:19:20.304041 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:19:20.538152 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:19:20.538119 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:19:20.796336 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:19:20.796310 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:19:21.442548 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:19:21.442484 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:19:22.001163 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:19:22.001133 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:19:22.326430 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:19:22.326385 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:19:28.717857 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:19:28.717825 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:20:14.911665 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:20:14.911631 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-m4c2p"] Jul 16 22:20:14.912122 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:20:14.911966 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea599498-2838-4ba4-b630-3bf6f13fb007" containerName="authorino" Jul 16 22:20:14.912122 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:20:14.911977 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea599498-2838-4ba4-b630-3bf6f13fb007" containerName="authorino" Jul 16 22:20:14.912122 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:20:14.912036 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea599498-2838-4ba4-b630-3bf6f13fb007" containerName="authorino" Jul 16 22:20:14.916046 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:20:14.916008 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-m4c2p" Jul 16 22:20:14.918868 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:20:14.918845 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 22:20:14.919094 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:20:14.918887 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\"" Jul 16 22:20:14.923421 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:20:14.923400 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-m4c2p"] Jul 16 22:20:15.005513 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:20:15.005481 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b9a3a9-8640-4461-a036-bcd019dd5a26-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-m4c2p\" (UID: \"b8b9a3a9-8640-4461-a036-bcd019dd5a26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-m4c2p" Jul 16 22:20:15.106831 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:20:15.106802 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b9a3a9-8640-4461-a036-bcd019dd5a26-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-m4c2p\" (UID: \"b8b9a3a9-8640-4461-a036-bcd019dd5a26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-m4c2p" Jul 16 22:20:15.107376 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:20:15.107358 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b9a3a9-8640-4461-a036-bcd019dd5a26-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-m4c2p\" (UID: \"b8b9a3a9-8640-4461-a036-bcd019dd5a26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-m4c2p" Jul 16 22:20:15.229972 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:20:15.229904 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-m4c2p" Jul 16 22:20:15.364786 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:20:15.364762 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-m4c2p"] Jul 16 22:20:15.366698 ip-10-0-128-255 kubenswrapper[2585]: W0716 22:20:15.366654 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8b9a3a9_8640_4461_a036_bcd019dd5a26.slice/crio-8f9d76800dbf0ebeddbddcdbdb1ff74d39887e4014a20f1b1321711e4db74e51 WatchSource:0}: Error finding container 8f9d76800dbf0ebeddbddcdbdb1ff74d39887e4014a20f1b1321711e4db74e51: Status 404 returned error can't find the container with id 8f9d76800dbf0ebeddbddcdbdb1ff74d39887e4014a20f1b1321711e4db74e51 Jul 16 22:20:15.368532 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:20:15.368512 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 22:20:15.517201 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:20:15.517129 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-m4c2p" event={"ID":"b8b9a3a9-8640-4461-a036-bcd019dd5a26","Type":"ContainerStarted","Data":"8f9d76800dbf0ebeddbddcdbdb1ff74d39887e4014a20f1b1321711e4db74e51"} Jul 16 22:20:19.540918 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:20:19.540885 2585 generic.go:356] "Generic (PLEG): container finished" podID="b8b9a3a9-8640-4461-a036-bcd019dd5a26" containerID="c59061a2314aa3b6fe7ab9c8c2b86858803d79a33fe773c591ae9610f8b12c67" exitCode=7 Jul 16 22:20:19.541339 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:20:19.540974 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-m4c2p" event={"ID":"b8b9a3a9-8640-4461-a036-bcd019dd5a26","Type":"ContainerDied","Data":"c59061a2314aa3b6fe7ab9c8c2b86858803d79a33fe773c591ae9610f8b12c67"} Jul 16 22:20:19.541339 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:20:19.541131 2585 scope.go:117] "RemoveContainer" containerID="c59061a2314aa3b6fe7ab9c8c2b86858803d79a33fe773c591ae9610f8b12c67" Jul 16 22:20:20.548483 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:20:20.548452 2585 generic.go:356] "Generic (PLEG): container finished" podID="b8b9a3a9-8640-4461-a036-bcd019dd5a26" containerID="a1a0efc2df49b7ac014312ca4f3205333ddbd2e6995eb933556c96ef50e48eea" exitCode=7 Jul 16 22:20:20.548957 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:20:20.548497 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-m4c2p" event={"ID":"b8b9a3a9-8640-4461-a036-bcd019dd5a26","Type":"ContainerDied","Data":"a1a0efc2df49b7ac014312ca4f3205333ddbd2e6995eb933556c96ef50e48eea"} Jul 16 22:20:20.548957 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:20:20.548548 2585 scope.go:117] "RemoveContainer" containerID="c59061a2314aa3b6fe7ab9c8c2b86858803d79a33fe773c591ae9610f8b12c67" Jul 16 22:20:20.548957 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:20:20.548774 2585 scope.go:117] "RemoveContainer" containerID="a1a0efc2df49b7ac014312ca4f3205333ddbd2e6995eb933556c96ef50e48eea" Jul 16 22:20:20.549085 ip-10-0-128-255 kubenswrapper[2585]: E0716 22:20:20.549025 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-m4c2p_odh-ai-gateway-infra(b8b9a3a9-8640-4461-a036-bcd019dd5a26)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-m4c2p" podUID="b8b9a3a9-8640-4461-a036-bcd019dd5a26" Jul 16 22:20:21.555792 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:20:21.555764 2585 scope.go:117] "RemoveContainer" containerID="a1a0efc2df49b7ac014312ca4f3205333ddbd2e6995eb933556c96ef50e48eea" Jul 16 22:20:21.556161 ip-10-0-128-255 kubenswrapper[2585]: E0716 22:20:21.555933 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-m4c2p_odh-ai-gateway-infra(b8b9a3a9-8640-4461-a036-bcd019dd5a26)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-m4c2p" podUID="b8b9a3a9-8640-4461-a036-bcd019dd5a26" Jul 16 22:20:33.929714 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:20:33.929678 2585 scope.go:117] "RemoveContainer" containerID="a1a0efc2df49b7ac014312ca4f3205333ddbd2e6995eb933556c96ef50e48eea" Jul 16 22:20:34.625028 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:20:34.624996 2585 generic.go:356] "Generic (PLEG): container finished" podID="b8b9a3a9-8640-4461-a036-bcd019dd5a26" containerID="5cec365d03aa8403263e9dcd1594904090d012d8a5f0960fcee3ae4be29abe1b" exitCode=7 Jul 16 22:20:34.625191 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:20:34.625069 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-m4c2p" event={"ID":"b8b9a3a9-8640-4461-a036-bcd019dd5a26","Type":"ContainerDied","Data":"5cec365d03aa8403263e9dcd1594904090d012d8a5f0960fcee3ae4be29abe1b"} Jul 16 22:20:34.625191 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:20:34.625129 2585 scope.go:117] "RemoveContainer" containerID="a1a0efc2df49b7ac014312ca4f3205333ddbd2e6995eb933556c96ef50e48eea" Jul 16 22:20:34.625474 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:20:34.625448 2585 scope.go:117] "RemoveContainer" containerID="5cec365d03aa8403263e9dcd1594904090d012d8a5f0960fcee3ae4be29abe1b" Jul 16 22:20:34.625689 ip-10-0-128-255 kubenswrapper[2585]: E0716 22:20:34.625665 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-m4c2p_odh-ai-gateway-infra(b8b9a3a9-8640-4461-a036-bcd019dd5a26)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-m4c2p" podUID="b8b9a3a9-8640-4461-a036-bcd019dd5a26" Jul 16 22:20:35.654418 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:20:35.654383 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-m4c2p"] Jul 16 22:20:35.788826 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:20:35.788800 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-m4c2p" Jul 16 22:20:35.882615 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:20:35.882590 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b9a3a9-8640-4461-a036-bcd019dd5a26-maas-api-service-ca\") pod \"b8b9a3a9-8640-4461-a036-bcd019dd5a26\" (UID: \"b8b9a3a9-8640-4461-a036-bcd019dd5a26\") " Jul 16 22:20:35.882883 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:20:35.882860 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8b9a3a9-8640-4461-a036-bcd019dd5a26-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b8b9a3a9-8640-4461-a036-bcd019dd5a26" (UID: "b8b9a3a9-8640-4461-a036-bcd019dd5a26"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 22:20:35.984096 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:20:35.984070 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b9a3a9-8640-4461-a036-bcd019dd5a26-maas-api-service-ca\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:20:36.638335 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:20:36.638298 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-m4c2p" event={"ID":"b8b9a3a9-8640-4461-a036-bcd019dd5a26","Type":"ContainerDied","Data":"8f9d76800dbf0ebeddbddcdbdb1ff74d39887e4014a20f1b1321711e4db74e51"} Jul 16 22:20:36.638335 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:20:36.638323 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-m4c2p" Jul 16 22:20:36.638632 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:20:36.638367 2585 scope.go:117] "RemoveContainer" containerID="5cec365d03aa8403263e9dcd1594904090d012d8a5f0960fcee3ae4be29abe1b" Jul 16 22:20:36.654119 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:20:36.654098 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-m4c2p"] Jul 16 22:20:36.658242 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:20:36.658205 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-m4c2p"] Jul 16 22:20:37.936123 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:20:37.936084 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8b9a3a9-8640-4461-a036-bcd019dd5a26" path="/var/lib/kubelet/pods/b8b9a3a9-8640-4461-a036-bcd019dd5a26/volumes" Jul 16 22:22:32.583274 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:22:32.583237 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:22:45.465060 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:22:45.465022 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:22:45.569263 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:22:45.569208 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:22:47.086592 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:22:47.086554 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:22:47.196330 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:22:47.196292 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:25:25.133857 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:25:25.133778 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:25:25.207170 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:25:25.207139 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:25:27.992409 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:25:27.992356 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:25:31.876769 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:25:31.876731 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:25:32.024921 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:25:32.024870 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:25:32.276360 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:25:32.276275 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:25:32.471699 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:25:32.471657 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:25:33.593167 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:25:33.593133 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:25:33.689441 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:25:33.689411 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:25:33.806539 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:25:33.806496 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:25:34.166990 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:25:34.166958 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:25:34.602954 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:25:34.602904 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:25:34.705981 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:25:34.705951 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:25:41.759498 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:25:41.759467 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:25:41.861085 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:25:41.861054 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:25:42.206118 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:25:42.206085 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:25:43.636913 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:25:43.636884 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:25:43.737210 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:25:43.737181 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:25:45.237232 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:25:45.237182 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-znk6v"] Jul 16 22:25:45.237601 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:25:45.237538 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8b9a3a9-8640-4461-a036-bcd019dd5a26" containerName="revoke-keys" Jul 16 22:25:45.237601 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:25:45.237550 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8b9a3a9-8640-4461-a036-bcd019dd5a26" containerName="revoke-keys" Jul 16 22:25:45.237601 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:25:45.237566 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8b9a3a9-8640-4461-a036-bcd019dd5a26" containerName="revoke-keys" Jul 16 22:25:45.237601 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:25:45.237573 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8b9a3a9-8640-4461-a036-bcd019dd5a26" containerName="revoke-keys" Jul 16 22:25:45.237734 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:25:45.237624 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8b9a3a9-8640-4461-a036-bcd019dd5a26" containerName="revoke-keys" Jul 16 22:25:45.237734 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:25:45.237634 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8b9a3a9-8640-4461-a036-bcd019dd5a26" containerName="revoke-keys" Jul 16 22:25:45.240589 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:25:45.240562 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-znk6v" Jul 16 22:25:45.243709 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:25:45.243688 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 22:25:45.243825 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:25:45.243796 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\"" Jul 16 22:25:45.252891 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:25:45.252862 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-znk6v"] Jul 16 22:25:45.400887 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:25:45.400851 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56fb3d36-d56b-4467-a957-5606bb80e330-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-znk6v\" (UID: \"56fb3d36-d56b-4467-a957-5606bb80e330\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-znk6v" Jul 16 22:25:45.501615 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:25:45.501539 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56fb3d36-d56b-4467-a957-5606bb80e330-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-znk6v\" (UID: \"56fb3d36-d56b-4467-a957-5606bb80e330\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-znk6v" Jul 16 22:25:45.502151 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:25:45.502132 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56fb3d36-d56b-4467-a957-5606bb80e330-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-znk6v\" (UID: \"56fb3d36-d56b-4467-a957-5606bb80e330\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-znk6v" Jul 16 22:25:45.551598 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:25:45.551572 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-znk6v" Jul 16 22:25:45.682343 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:25:45.682311 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-znk6v"] Jul 16 22:25:45.684235 ip-10-0-128-255 kubenswrapper[2585]: W0716 22:25:45.684192 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56fb3d36_d56b_4467_a957_5606bb80e330.slice/crio-2de90a8963656418cd34e3190d80ecb56b1631187542ba5869876172516b2212 WatchSource:0}: Error finding container 2de90a8963656418cd34e3190d80ecb56b1631187542ba5869876172516b2212: Status 404 returned error can't find the container with id 2de90a8963656418cd34e3190d80ecb56b1631187542ba5869876172516b2212 Jul 16 22:25:45.772879 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:25:45.772811 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:25:46.169623 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:25:46.169590 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-znk6v" event={"ID":"56fb3d36-d56b-4467-a957-5606bb80e330","Type":"ContainerStarted","Data":"6230ab7f67c1a8866b237eef4b2a70ba183ed7a06b87d0261fb785e754eff537"} Jul 16 22:25:46.169755 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:25:46.169649 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-znk6v" event={"ID":"56fb3d36-d56b-4467-a957-5606bb80e330","Type":"ContainerStarted","Data":"2de90a8963656418cd34e3190d80ecb56b1631187542ba5869876172516b2212"} Jul 16 22:25:46.193362 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:25:46.189935 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-znk6v" podStartSLOduration=1.189910147 podStartE2EDuration="1.189910147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:25:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:25:46.187061093 +0000 UTC m=+2294.809504055" watchObservedRunningTime="2026-07-16 22:25:46.189910147 +0000 UTC m=+2294.812353104" Jul 16 22:25:47.175327 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:25:47.175295 2585 generic.go:356] "Generic (PLEG): container finished" podID="56fb3d36-d56b-4467-a957-5606bb80e330" containerID="6230ab7f67c1a8866b237eef4b2a70ba183ed7a06b87d0261fb785e754eff537" exitCode=6 Jul 16 22:25:47.175716 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:25:47.175377 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-znk6v" event={"ID":"56fb3d36-d56b-4467-a957-5606bb80e330","Type":"ContainerDied","Data":"6230ab7f67c1a8866b237eef4b2a70ba183ed7a06b87d0261fb785e754eff537"} Jul 16 22:25:47.175716 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:25:47.175499 2585 scope.go:117] "RemoveContainer" containerID="6230ab7f67c1a8866b237eef4b2a70ba183ed7a06b87d0261fb785e754eff537" Jul 16 22:25:48.181907 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:25:48.181868 2585 generic.go:356] "Generic (PLEG): container finished" podID="56fb3d36-d56b-4467-a957-5606bb80e330" containerID="ae4d2884452314be629058c8ce4d7af3c916d96414b8b3407ff5dcf9394c8206" exitCode=6 Jul 16 22:25:48.182347 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:25:48.181954 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-znk6v" event={"ID":"56fb3d36-d56b-4467-a957-5606bb80e330","Type":"ContainerDied","Data":"ae4d2884452314be629058c8ce4d7af3c916d96414b8b3407ff5dcf9394c8206"} Jul 16 22:25:48.182347 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:25:48.182011 2585 scope.go:117] "RemoveContainer" containerID="6230ab7f67c1a8866b237eef4b2a70ba183ed7a06b87d0261fb785e754eff537" Jul 16 22:25:48.182347 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:25:48.182193 2585 scope.go:117] "RemoveContainer" containerID="ae4d2884452314be629058c8ce4d7af3c916d96414b8b3407ff5dcf9394c8206" Jul 16 22:25:48.182465 ip-10-0-128-255 kubenswrapper[2585]: E0716 22:25:48.182440 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-znk6v_odh-ai-gateway-infra(56fb3d36-d56b-4467-a957-5606bb80e330)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-znk6v" podUID="56fb3d36-d56b-4467-a957-5606bb80e330" Jul 16 22:25:49.188292 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:25:49.188263 2585 scope.go:117] "RemoveContainer" containerID="ae4d2884452314be629058c8ce4d7af3c916d96414b8b3407ff5dcf9394c8206" Jul 16 22:25:49.188714 ip-10-0-128-255 kubenswrapper[2585]: E0716 22:25:49.188450 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-znk6v_odh-ai-gateway-infra(56fb3d36-d56b-4467-a957-5606bb80e330)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-znk6v" podUID="56fb3d36-d56b-4467-a957-5606bb80e330" Jul 16 22:26:03.929387 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:03.929355 2585 scope.go:117] "RemoveContainer" containerID="ae4d2884452314be629058c8ce4d7af3c916d96414b8b3407ff5dcf9394c8206" Jul 16 22:26:04.263876 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:04.263847 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-znk6v" event={"ID":"56fb3d36-d56b-4467-a957-5606bb80e330","Type":"ContainerStarted","Data":"88dd5c90eee08c90c767b3d0baf2d874168bf7a023bff163aa2ff1a9d4c9cd9e"} Jul 16 22:26:05.269976 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:05.269940 2585 generic.go:356] "Generic (PLEG): container finished" podID="56fb3d36-d56b-4467-a957-5606bb80e330" containerID="88dd5c90eee08c90c767b3d0baf2d874168bf7a023bff163aa2ff1a9d4c9cd9e" exitCode=6 Jul 16 22:26:05.270415 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:05.270017 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-znk6v" event={"ID":"56fb3d36-d56b-4467-a957-5606bb80e330","Type":"ContainerDied","Data":"88dd5c90eee08c90c767b3d0baf2d874168bf7a023bff163aa2ff1a9d4c9cd9e"} Jul 16 22:26:05.270415 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:05.270081 2585 scope.go:117] "RemoveContainer" containerID="ae4d2884452314be629058c8ce4d7af3c916d96414b8b3407ff5dcf9394c8206" Jul 16 22:26:05.270415 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:05.270361 2585 scope.go:117] "RemoveContainer" containerID="88dd5c90eee08c90c767b3d0baf2d874168bf7a023bff163aa2ff1a9d4c9cd9e" Jul 16 22:26:05.270584 ip-10-0-128-255 kubenswrapper[2585]: E0716 22:26:05.270561 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-znk6v_odh-ai-gateway-infra(56fb3d36-d56b-4467-a957-5606bb80e330)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-znk6v" podUID="56fb3d36-d56b-4467-a957-5606bb80e330" Jul 16 22:26:05.296209 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:05.296175 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-znk6v"] Jul 16 22:26:06.407856 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:06.407835 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-znk6v" Jul 16 22:26:06.474921 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:06.474895 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56fb3d36-d56b-4467-a957-5606bb80e330-maas-api-service-ca\") pod \"56fb3d36-d56b-4467-a957-5606bb80e330\" (UID: \"56fb3d36-d56b-4467-a957-5606bb80e330\") " Jul 16 22:26:06.475256 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:06.475234 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56fb3d36-d56b-4467-a957-5606bb80e330-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "56fb3d36-d56b-4467-a957-5606bb80e330" (UID: "56fb3d36-d56b-4467-a957-5606bb80e330"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 22:26:06.576274 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:06.576247 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56fb3d36-d56b-4467-a957-5606bb80e330-maas-api-service-ca\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:26:07.282525 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:07.282491 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-znk6v" Jul 16 22:26:07.282672 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:07.282531 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-znk6v" event={"ID":"56fb3d36-d56b-4467-a957-5606bb80e330","Type":"ContainerDied","Data":"2de90a8963656418cd34e3190d80ecb56b1631187542ba5869876172516b2212"} Jul 16 22:26:07.282672 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:07.282590 2585 scope.go:117] "RemoveContainer" containerID="88dd5c90eee08c90c767b3d0baf2d874168bf7a023bff163aa2ff1a9d4c9cd9e" Jul 16 22:26:07.308607 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:07.308577 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-znk6v"] Jul 16 22:26:07.314645 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:07.314625 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-znk6v"] Jul 16 22:26:07.935035 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:07.935000 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56fb3d36-d56b-4467-a957-5606bb80e330" path="/var/lib/kubelet/pods/56fb3d36-d56b-4467-a957-5606bb80e330/volumes" Jul 16 22:26:27.065837 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:27.065749 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:26:36.680327 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:36.680291 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-jnqcb"] Jul 16 22:26:36.680800 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:36.680608 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56fb3d36-d56b-4467-a957-5606bb80e330" containerName="revoke-keys" Jul 16 22:26:36.680800 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:36.680619 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="56fb3d36-d56b-4467-a957-5606bb80e330" containerName="revoke-keys" Jul 16 22:26:36.680800 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:36.680634 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8b9a3a9-8640-4461-a036-bcd019dd5a26" containerName="revoke-keys" Jul 16 22:26:36.680800 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:36.680639 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8b9a3a9-8640-4461-a036-bcd019dd5a26" containerName="revoke-keys" Jul 16 22:26:36.680800 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:36.680662 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56fb3d36-d56b-4467-a957-5606bb80e330" containerName="revoke-keys" Jul 16 22:26:36.680800 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:36.680668 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="56fb3d36-d56b-4467-a957-5606bb80e330" containerName="revoke-keys" Jul 16 22:26:36.680800 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:36.680675 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56fb3d36-d56b-4467-a957-5606bb80e330" containerName="revoke-keys" Jul 16 22:26:36.680800 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:36.680680 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="56fb3d36-d56b-4467-a957-5606bb80e330" containerName="revoke-keys" Jul 16 22:26:36.680800 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:36.680739 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="56fb3d36-d56b-4467-a957-5606bb80e330" containerName="revoke-keys" Jul 16 22:26:36.680800 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:36.680746 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="56fb3d36-d56b-4467-a957-5606bb80e330" containerName="revoke-keys" Jul 16 22:26:36.680800 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:36.680752 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8b9a3a9-8640-4461-a036-bcd019dd5a26" containerName="revoke-keys" Jul 16 22:26:36.685038 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:36.685020 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-jnqcb" Jul 16 22:26:36.687896 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:36.687872 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\"" Jul 16 22:26:36.688758 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:36.688741 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 22:26:36.697855 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:36.697831 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-jnqcb"] Jul 16 22:26:36.831467 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:36.831434 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52ead3dd-583d-4ad2-be6d-a97fbaf19894-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-jnqcb\" (UID: \"52ead3dd-583d-4ad2-be6d-a97fbaf19894\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-jnqcb" Jul 16 22:26:36.932831 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:36.932756 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52ead3dd-583d-4ad2-be6d-a97fbaf19894-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-jnqcb\" (UID: \"52ead3dd-583d-4ad2-be6d-a97fbaf19894\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-jnqcb" Jul 16 22:26:36.933432 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:36.933406 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52ead3dd-583d-4ad2-be6d-a97fbaf19894-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-jnqcb\" (UID: \"52ead3dd-583d-4ad2-be6d-a97fbaf19894\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-jnqcb" Jul 16 22:26:36.994830 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:36.994805 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-jnqcb" Jul 16 22:26:37.130021 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:37.129994 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-jnqcb"] Jul 16 22:26:37.131854 ip-10-0-128-255 kubenswrapper[2585]: W0716 22:26:37.131821 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52ead3dd_583d_4ad2_be6d_a97fbaf19894.slice/crio-162698506ce8892054cfee25b3d6010c03f5dfd555c9b86ee520de138252486a WatchSource:0}: Error finding container 162698506ce8892054cfee25b3d6010c03f5dfd555c9b86ee520de138252486a: Status 404 returned error can't find the container with id 162698506ce8892054cfee25b3d6010c03f5dfd555c9b86ee520de138252486a Jul 16 22:26:37.428350 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:37.428318 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-jnqcb" event={"ID":"52ead3dd-583d-4ad2-be6d-a97fbaf19894","Type":"ContainerStarted","Data":"ae61f2f1fb963c324aaa3b2d9795420d15e51330a0ff539219bbd206b9d8f2e0"} Jul 16 22:26:37.428536 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:37.428369 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-jnqcb" event={"ID":"52ead3dd-583d-4ad2-be6d-a97fbaf19894","Type":"ContainerStarted","Data":"162698506ce8892054cfee25b3d6010c03f5dfd555c9b86ee520de138252486a"} Jul 16 22:26:37.446858 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:37.446813 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-jnqcb" podStartSLOduration=1.446800309 podStartE2EDuration="1.446800309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:26:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:26:37.445296168 +0000 UTC m=+2346.067739120" watchObservedRunningTime="2026-07-16 22:26:37.446800309 +0000 UTC m=+2346.069243262" Jul 16 22:26:38.434948 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:38.434911 2585 generic.go:356] "Generic (PLEG): container finished" podID="52ead3dd-583d-4ad2-be6d-a97fbaf19894" containerID="ae61f2f1fb963c324aaa3b2d9795420d15e51330a0ff539219bbd206b9d8f2e0" exitCode=6 Jul 16 22:26:38.435360 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:38.434998 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-jnqcb" event={"ID":"52ead3dd-583d-4ad2-be6d-a97fbaf19894","Type":"ContainerDied","Data":"ae61f2f1fb963c324aaa3b2d9795420d15e51330a0ff539219bbd206b9d8f2e0"} Jul 16 22:26:38.435360 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:38.435148 2585 scope.go:117] "RemoveContainer" containerID="ae61f2f1fb963c324aaa3b2d9795420d15e51330a0ff539219bbd206b9d8f2e0" Jul 16 22:26:38.857377 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:38.857352 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:26:39.442058 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:39.442026 2585 generic.go:356] "Generic (PLEG): container finished" podID="52ead3dd-583d-4ad2-be6d-a97fbaf19894" containerID="6273f8887b2e3b06a5d05cd5686087bdd88fda37c4093e7cb1fd7b1199dacd42" exitCode=6 Jul 16 22:26:39.442547 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:39.442111 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-jnqcb" event={"ID":"52ead3dd-583d-4ad2-be6d-a97fbaf19894","Type":"ContainerDied","Data":"6273f8887b2e3b06a5d05cd5686087bdd88fda37c4093e7cb1fd7b1199dacd42"} Jul 16 22:26:39.442547 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:39.442185 2585 scope.go:117] "RemoveContainer" containerID="ae61f2f1fb963c324aaa3b2d9795420d15e51330a0ff539219bbd206b9d8f2e0" Jul 16 22:26:39.442547 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:39.442394 2585 scope.go:117] "RemoveContainer" containerID="6273f8887b2e3b06a5d05cd5686087bdd88fda37c4093e7cb1fd7b1199dacd42" Jul 16 22:26:39.442686 ip-10-0-128-255 kubenswrapper[2585]: E0716 22:26:39.442585 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-jnqcb_odh-ai-gateway-infra(52ead3dd-583d-4ad2-be6d-a97fbaf19894)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-jnqcb" podUID="52ead3dd-583d-4ad2-be6d-a97fbaf19894" Jul 16 22:26:40.448639 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:40.448610 2585 scope.go:117] "RemoveContainer" containerID="6273f8887b2e3b06a5d05cd5686087bdd88fda37c4093e7cb1fd7b1199dacd42" Jul 16 22:26:40.449021 ip-10-0-128-255 kubenswrapper[2585]: E0716 22:26:40.448822 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-jnqcb_odh-ai-gateway-infra(52ead3dd-583d-4ad2-be6d-a97fbaf19894)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-jnqcb" podUID="52ead3dd-583d-4ad2-be6d-a97fbaf19894" Jul 16 22:26:47.856401 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:47.856364 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:26:55.929090 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:55.929054 2585 scope.go:117] "RemoveContainer" containerID="6273f8887b2e3b06a5d05cd5686087bdd88fda37c4093e7cb1fd7b1199dacd42" Jul 16 22:26:56.526655 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:56.526627 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-jnqcb" event={"ID":"52ead3dd-583d-4ad2-be6d-a97fbaf19894","Type":"ContainerStarted","Data":"3c5c936bb7f527167e4534af3aefe68dd409341386252058130591f78e171eb0"} Jul 16 22:26:57.535471 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:57.535435 2585 generic.go:356] "Generic (PLEG): container finished" podID="52ead3dd-583d-4ad2-be6d-a97fbaf19894" containerID="3c5c936bb7f527167e4534af3aefe68dd409341386252058130591f78e171eb0" exitCode=6 Jul 16 22:26:57.535896 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:57.535498 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-jnqcb" event={"ID":"52ead3dd-583d-4ad2-be6d-a97fbaf19894","Type":"ContainerDied","Data":"3c5c936bb7f527167e4534af3aefe68dd409341386252058130591f78e171eb0"} Jul 16 22:26:57.535896 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:57.535551 2585 scope.go:117] "RemoveContainer" containerID="6273f8887b2e3b06a5d05cd5686087bdd88fda37c4093e7cb1fd7b1199dacd42" Jul 16 22:26:57.535896 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:57.535827 2585 scope.go:117] "RemoveContainer" containerID="3c5c936bb7f527167e4534af3aefe68dd409341386252058130591f78e171eb0" Jul 16 22:26:57.536047 ip-10-0-128-255 kubenswrapper[2585]: E0716 22:26:57.536026 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-jnqcb_odh-ai-gateway-infra(52ead3dd-583d-4ad2-be6d-a97fbaf19894)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-jnqcb" podUID="52ead3dd-583d-4ad2-be6d-a97fbaf19894" Jul 16 22:26:57.552917 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:57.552891 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-jnqcb"] Jul 16 22:26:58.672753 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:58.672730 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-jnqcb" Jul 16 22:26:58.696531 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:58.696498 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52ead3dd-583d-4ad2-be6d-a97fbaf19894-maas-api-service-ca\") pod \"52ead3dd-583d-4ad2-be6d-a97fbaf19894\" (UID: \"52ead3dd-583d-4ad2-be6d-a97fbaf19894\") " Jul 16 22:26:58.696822 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:58.696801 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52ead3dd-583d-4ad2-be6d-a97fbaf19894-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "52ead3dd-583d-4ad2-be6d-a97fbaf19894" (UID: "52ead3dd-583d-4ad2-be6d-a97fbaf19894"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 22:26:58.797917 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:58.797895 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52ead3dd-583d-4ad2-be6d-a97fbaf19894-maas-api-service-ca\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:26:59.549076 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:59.549044 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-jnqcb" event={"ID":"52ead3dd-583d-4ad2-be6d-a97fbaf19894","Type":"ContainerDied","Data":"162698506ce8892054cfee25b3d6010c03f5dfd555c9b86ee520de138252486a"} Jul 16 22:26:59.549076 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:59.549074 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-jnqcb" Jul 16 22:26:59.549338 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:59.549102 2585 scope.go:117] "RemoveContainer" containerID="3c5c936bb7f527167e4534af3aefe68dd409341386252058130591f78e171eb0" Jul 16 22:26:59.575699 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:59.575677 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-jnqcb"] Jul 16 22:26:59.585450 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:59.585422 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-jnqcb"] Jul 16 22:26:59.934134 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:26:59.934058 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52ead3dd-583d-4ad2-be6d-a97fbaf19894" path="/var/lib/kubelet/pods/52ead3dd-583d-4ad2-be6d-a97fbaf19894/volumes" Jul 16 22:27:46.331424 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:27:46.331384 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:27:48.158521 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:27:48.158482 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:27:48.756166 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:27:48.756135 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:27:48.856603 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:27:48.856569 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:28:35.376433 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:35.376398 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-bc6tl"] Jul 16 22:28:35.376917 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:35.376762 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52ead3dd-583d-4ad2-be6d-a97fbaf19894" containerName="revoke-keys" Jul 16 22:28:35.376917 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:35.376774 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="52ead3dd-583d-4ad2-be6d-a97fbaf19894" containerName="revoke-keys" Jul 16 22:28:35.376917 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:35.376791 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52ead3dd-583d-4ad2-be6d-a97fbaf19894" containerName="revoke-keys" Jul 16 22:28:35.376917 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:35.376797 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="52ead3dd-583d-4ad2-be6d-a97fbaf19894" containerName="revoke-keys" Jul 16 22:28:35.376917 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:35.376804 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52ead3dd-583d-4ad2-be6d-a97fbaf19894" containerName="revoke-keys" Jul 16 22:28:35.376917 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:35.376810 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="52ead3dd-583d-4ad2-be6d-a97fbaf19894" containerName="revoke-keys" Jul 16 22:28:35.376917 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:35.376861 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="56fb3d36-d56b-4467-a957-5606bb80e330" containerName="revoke-keys" Jul 16 22:28:35.376917 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:35.376869 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="52ead3dd-583d-4ad2-be6d-a97fbaf19894" containerName="revoke-keys" Jul 16 22:28:35.376917 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:35.376879 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="52ead3dd-583d-4ad2-be6d-a97fbaf19894" containerName="revoke-keys" Jul 16 22:28:35.379659 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:35.379643 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-bc6tl" Jul 16 22:28:35.381919 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:35.381890 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 22:28:35.382071 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:35.382025 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\"" Jul 16 22:28:35.387643 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:35.387616 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-bc6tl"] Jul 16 22:28:35.412833 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:35.412806 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/861ea66c-1923-4fde-a011-660b5dbd94b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-bc6tl\" (UID: \"861ea66c-1923-4fde-a011-660b5dbd94b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-bc6tl" Jul 16 22:28:35.513405 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:35.513368 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/861ea66c-1923-4fde-a011-660b5dbd94b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-bc6tl\" (UID: \"861ea66c-1923-4fde-a011-660b5dbd94b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-bc6tl" Jul 16 22:28:35.514019 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:35.513999 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/861ea66c-1923-4fde-a011-660b5dbd94b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-bc6tl\" (UID: \"861ea66c-1923-4fde-a011-660b5dbd94b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-bc6tl" Jul 16 22:28:35.691172 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:35.691099 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-bc6tl" Jul 16 22:28:36.026454 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:36.026430 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-bc6tl"] Jul 16 22:28:36.028842 ip-10-0-128-255 kubenswrapper[2585]: W0716 22:28:36.028807 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod861ea66c_1923_4fde_a011_660b5dbd94b4.slice/crio-9ce520b771cfe68616b723bec9f423809517391aad53cce5de5a9e93b4b5e73d WatchSource:0}: Error finding container 9ce520b771cfe68616b723bec9f423809517391aad53cce5de5a9e93b4b5e73d: Status 404 returned error can't find the container with id 9ce520b771cfe68616b723bec9f423809517391aad53cce5de5a9e93b4b5e73d Jul 16 22:28:37.040709 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:37.040616 2585 generic.go:356] "Generic (PLEG): container finished" podID="861ea66c-1923-4fde-a011-660b5dbd94b4" containerID="599ccb15ad1ccebe13c43c4bd2a1a19672ab0a9b4904f79ebbd3197af3fb4e49" exitCode=6 Jul 16 22:28:37.040709 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:37.040668 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-bc6tl" event={"ID":"861ea66c-1923-4fde-a011-660b5dbd94b4","Type":"ContainerDied","Data":"599ccb15ad1ccebe13c43c4bd2a1a19672ab0a9b4904f79ebbd3197af3fb4e49"} Jul 16 22:28:37.040709 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:37.040704 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-bc6tl" event={"ID":"861ea66c-1923-4fde-a011-660b5dbd94b4","Type":"ContainerStarted","Data":"9ce520b771cfe68616b723bec9f423809517391aad53cce5de5a9e93b4b5e73d"} Jul 16 22:28:37.041312 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:37.040833 2585 scope.go:117] "RemoveContainer" containerID="599ccb15ad1ccebe13c43c4bd2a1a19672ab0a9b4904f79ebbd3197af3fb4e49" Jul 16 22:28:38.029111 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:38.029030 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:28:38.054098 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:38.054061 2585 generic.go:356] "Generic (PLEG): container finished" podID="861ea66c-1923-4fde-a011-660b5dbd94b4" containerID="87a373649139da595500f24c7c67ed0f627912fba5f869f7d04df8b7488df489" exitCode=6 Jul 16 22:28:38.054569 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:38.054137 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-bc6tl" event={"ID":"861ea66c-1923-4fde-a011-660b5dbd94b4","Type":"ContainerDied","Data":"87a373649139da595500f24c7c67ed0f627912fba5f869f7d04df8b7488df489"} Jul 16 22:28:38.054569 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:38.054235 2585 scope.go:117] "RemoveContainer" containerID="599ccb15ad1ccebe13c43c4bd2a1a19672ab0a9b4904f79ebbd3197af3fb4e49" Jul 16 22:28:38.054569 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:38.054407 2585 scope.go:117] "RemoveContainer" containerID="87a373649139da595500f24c7c67ed0f627912fba5f869f7d04df8b7488df489" Jul 16 22:28:38.054704 ip-10-0-128-255 kubenswrapper[2585]: E0716 22:28:38.054578 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-bc6tl_odh-ai-gateway-infra(861ea66c-1923-4fde-a011-660b5dbd94b4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-bc6tl" podUID="861ea66c-1923-4fde-a011-660b5dbd94b4" Jul 16 22:28:38.141906 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:38.141876 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:28:39.061016 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:39.060985 2585 scope.go:117] "RemoveContainer" containerID="87a373649139da595500f24c7c67ed0f627912fba5f869f7d04df8b7488df489" Jul 16 22:28:39.061461 ip-10-0-128-255 kubenswrapper[2585]: E0716 22:28:39.061198 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-bc6tl_odh-ai-gateway-infra(861ea66c-1923-4fde-a011-660b5dbd94b4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-bc6tl" podUID="861ea66c-1923-4fde-a011-660b5dbd94b4" Jul 16 22:28:40.803070 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:40.803036 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:28:44.158920 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:44.158890 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:28:44.531538 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:44.531428 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:28:50.069599 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:50.069562 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:28:50.143370 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:50.143270 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:28:50.601598 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:50.601528 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:28:50.694114 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:50.694087 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:28:50.793687 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:50.793657 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:28:50.929506 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:50.929444 2585 scope.go:117] "RemoveContainer" containerID="87a373649139da595500f24c7c67ed0f627912fba5f869f7d04df8b7488df489" Jul 16 22:28:52.127430 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:52.127401 2585 generic.go:356] "Generic (PLEG): container finished" podID="861ea66c-1923-4fde-a011-660b5dbd94b4" containerID="77d97a3a17bc1d3da09c10634922c30e75b7db0bbbeb84759173d24bb25c7a49" exitCode=6 Jul 16 22:28:52.127813 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:52.127481 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-bc6tl" event={"ID":"861ea66c-1923-4fde-a011-660b5dbd94b4","Type":"ContainerDied","Data":"77d97a3a17bc1d3da09c10634922c30e75b7db0bbbeb84759173d24bb25c7a49"} Jul 16 22:28:52.127813 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:52.127544 2585 scope.go:117] "RemoveContainer" containerID="87a373649139da595500f24c7c67ed0f627912fba5f869f7d04df8b7488df489" Jul 16 22:28:52.127813 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:52.127812 2585 scope.go:117] "RemoveContainer" containerID="77d97a3a17bc1d3da09c10634922c30e75b7db0bbbeb84759173d24bb25c7a49" Jul 16 22:28:52.128031 ip-10-0-128-255 kubenswrapper[2585]: E0716 22:28:52.128013 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-bc6tl_odh-ai-gateway-infra(861ea66c-1923-4fde-a011-660b5dbd94b4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-bc6tl" podUID="861ea66c-1923-4fde-a011-660b5dbd94b4" Jul 16 22:28:52.231616 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:52.231578 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:28:53.153483 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:53.153450 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-bc6tl"] Jul 16 22:28:53.283783 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:53.283761 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-bc6tl" Jul 16 22:28:53.359050 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:53.359025 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/861ea66c-1923-4fde-a011-660b5dbd94b4-maas-api-service-ca\") pod \"861ea66c-1923-4fde-a011-660b5dbd94b4\" (UID: \"861ea66c-1923-4fde-a011-660b5dbd94b4\") " Jul 16 22:28:53.359371 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:53.359346 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/861ea66c-1923-4fde-a011-660b5dbd94b4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "861ea66c-1923-4fde-a011-660b5dbd94b4" (UID: "861ea66c-1923-4fde-a011-660b5dbd94b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 22:28:53.459880 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:53.459811 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/861ea66c-1923-4fde-a011-660b5dbd94b4-maas-api-service-ca\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:28:54.140005 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:54.139974 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-bc6tl" Jul 16 22:28:54.140005 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:54.139985 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-bc6tl" event={"ID":"861ea66c-1923-4fde-a011-660b5dbd94b4","Type":"ContainerDied","Data":"9ce520b771cfe68616b723bec9f423809517391aad53cce5de5a9e93b4b5e73d"} Jul 16 22:28:54.140243 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:54.140046 2585 scope.go:117] "RemoveContainer" containerID="77d97a3a17bc1d3da09c10634922c30e75b7db0bbbeb84759173d24bb25c7a49" Jul 16 22:28:54.161687 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:54.161660 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-bc6tl"] Jul 16 22:28:54.168770 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:54.168743 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-bc6tl"] Jul 16 22:28:55.934743 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:55.934707 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="861ea66c-1923-4fde-a011-660b5dbd94b4" path="/var/lib/kubelet/pods/861ea66c-1923-4fde-a011-660b5dbd94b4/volumes" Jul 16 22:28:58.042394 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:58.042349 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:28:58.156880 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:58.156846 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:28:58.293408 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:58.293326 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:28:58.538490 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:58.538459 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:28:58.863129 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:28:58.863093 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:29:02.484044 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:29:02.484007 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:29:04.976627 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:29:04.976596 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:29:24.860558 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:29:24.860480 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:29:31.023935 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:29:31.023906 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:29:31.178578 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:29:31.178546 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:29:31.278628 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:29:31.278555 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:29:47.499476 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:29:47.499442 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:29:53.802850 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:29:53.802818 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:29:53.928667 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:29:53.928633 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:29:54.056780 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:29:54.056699 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:30:04.575361 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:30:04.575329 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:30:05.389773 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:30:05.389742 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:30:11.825024 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:30:11.824991 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:30:12.001763 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:30:12.001723 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:30:12.232648 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:30:12.232565 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:30:12.330583 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:30:12.330551 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:30:12.979442 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:30:12.979412 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:30:13.108780 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:30:13.108754 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:30:13.196501 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:30:13.196473 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:30:14.542816 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:30:14.542778 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:30:14.987584 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:30:14.987558 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:30:21.214545 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:30:21.214507 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:30:21.314369 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:30:21.314333 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:30:21.897805 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:30:21.897772 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:30:22.321171 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:30:22.321143 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:30:29.835413 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:30:29.835375 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:30:29.966501 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:30:29.966470 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:30:30.122057 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:30:30.121975 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:30:30.326971 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:30:30.326930 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:30:30.672454 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:30:30.672427 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:30:30.991444 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:30:30.991365 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:30:34.306128 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:30:34.306093 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:30:34.497638 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:30:34.497609 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:30:34.665840 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:30:34.665763 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:30:34.843489 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:30:34.843460 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:30:35.459876 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:30:35.459839 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:30:35.582823 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:30:35.582792 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:30:38.680095 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:30:38.680060 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:30:39.334677 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:30:39.334645 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:30:39.434939 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:30:39.434911 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:30:39.752394 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:30:39.752323 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:30:41.621927 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:30:41.621887 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:30:46.324374 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:30:46.324331 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:30:46.606740 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:30:46.606667 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:30:51.517768 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:30:51.517735 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:30:51.633665 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:30:51.633637 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:30:57.479893 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:30:57.479817 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:31:00.305786 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:00.305753 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:31:01.874153 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:01.874119 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:31:07.911601 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:07.911568 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:31:07.979343 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:07.979313 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:31:12.585849 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:12.585820 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:31:12.813520 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:12.813459 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:31:12.943185 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:12.943103 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:31:13.040620 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:13.040584 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:31:13.154868 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:13.154838 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:31:19.962087 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:19.962057 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:31:20.069717 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:20.069680 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:31:20.249570 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:20.249481 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:31:20.378580 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:20.378551 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:31:28.277398 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:28.277368 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:31:28.550722 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:28.550692 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:31:37.590289 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:37.590255 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:31:38.093816 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:38.093772 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:31:38.389610 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:38.389532 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:31:41.773231 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:41.773194 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:31:42.092978 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:42.092953 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:31:42.393012 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:42.392926 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:31:42.591173 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:42.591138 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:31:43.365295 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:43.365193 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:31:43.521703 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:43.521671 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:31:46.167315 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:46.167280 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:31:48.134641 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:48.134609 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:31:56.820255 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:56.820207 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-601103b4/e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr"] Jul 16 22:31:56.820702 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:56.820551 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="861ea66c-1923-4fde-a011-660b5dbd94b4" containerName="revoke-keys" Jul 16 22:31:56.820702 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:56.820562 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="861ea66c-1923-4fde-a011-660b5dbd94b4" containerName="revoke-keys" Jul 16 22:31:56.820702 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:56.820579 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="861ea66c-1923-4fde-a011-660b5dbd94b4" containerName="revoke-keys" Jul 16 22:31:56.820702 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:56.820585 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="861ea66c-1923-4fde-a011-660b5dbd94b4" containerName="revoke-keys" Jul 16 22:31:56.820702 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:56.820643 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="861ea66c-1923-4fde-a011-660b5dbd94b4" containerName="revoke-keys" Jul 16 22:31:56.820702 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:56.820650 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="861ea66c-1923-4fde-a011-660b5dbd94b4" containerName="revoke-keys" Jul 16 22:31:56.820702 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:56.820657 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="52ead3dd-583d-4ad2-be6d-a97fbaf19894" containerName="revoke-keys" Jul 16 22:31:56.823770 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:56.823753 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-601103b4/e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr" Jul 16 22:31:56.832769 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:56.832747 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-601103b4\"/\"kube-root-ca.crt\"" Jul 16 22:31:56.832877 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:56.832818 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-601103b4\"/\"e2e-lifecycle-model-601103b4-kserve-self-signed-certs\"" Jul 16 22:31:56.833041 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:56.833028 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-601103b4\"/\"default-dockercfg-s7bsn\"" Jul 16 22:31:56.833095 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:56.833077 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-601103b4\"/\"openshift-service-ca.crt\"" Jul 16 22:31:56.851918 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:56.851890 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-601103b4/e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr"] Jul 16 22:31:56.876346 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:56.876317 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:31:56.958735 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:56.958696 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b89fb2b1-14a7-472e-8b1f-5a7a5b761d52-tls-certs\") pod \"e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr\" (UID: \"b89fb2b1-14a7-472e-8b1f-5a7a5b761d52\") " pod="ai-tenant-e2e-mt-601103b4/e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr" Jul 16 22:31:56.958905 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:56.958768 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b89fb2b1-14a7-472e-8b1f-5a7a5b761d52-home\") pod \"e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr\" (UID: \"b89fb2b1-14a7-472e-8b1f-5a7a5b761d52\") " pod="ai-tenant-e2e-mt-601103b4/e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr" Jul 16 22:31:56.958905 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:56.958805 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfj7d\" (UniqueName: \"kubernetes.io/projected/b89fb2b1-14a7-472e-8b1f-5a7a5b761d52-kube-api-access-tfj7d\") pod \"e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr\" (UID: \"b89fb2b1-14a7-472e-8b1f-5a7a5b761d52\") " pod="ai-tenant-e2e-mt-601103b4/e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr" Jul 16 22:31:56.958905 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:56.958861 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b89fb2b1-14a7-472e-8b1f-5a7a5b761d52-tmp-dir\") pod \"e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr\" (UID: \"b89fb2b1-14a7-472e-8b1f-5a7a5b761d52\") " pod="ai-tenant-e2e-mt-601103b4/e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr" Jul 16 22:31:56.958905 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:56.958899 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b89fb2b1-14a7-472e-8b1f-5a7a5b761d52-dshm\") pod \"e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr\" (UID: \"b89fb2b1-14a7-472e-8b1f-5a7a5b761d52\") " pod="ai-tenant-e2e-mt-601103b4/e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr" Jul 16 22:31:56.959090 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:56.958932 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b89fb2b1-14a7-472e-8b1f-5a7a5b761d52-model-cache\") pod \"e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr\" (UID: \"b89fb2b1-14a7-472e-8b1f-5a7a5b761d52\") " pod="ai-tenant-e2e-mt-601103b4/e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr" Jul 16 22:31:57.059932 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:57.059891 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b89fb2b1-14a7-472e-8b1f-5a7a5b761d52-model-cache\") pod \"e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr\" (UID: \"b89fb2b1-14a7-472e-8b1f-5a7a5b761d52\") " pod="ai-tenant-e2e-mt-601103b4/e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr" Jul 16 22:31:57.060120 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:57.059972 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b89fb2b1-14a7-472e-8b1f-5a7a5b761d52-tls-certs\") pod \"e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr\" (UID: \"b89fb2b1-14a7-472e-8b1f-5a7a5b761d52\") " pod="ai-tenant-e2e-mt-601103b4/e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr" Jul 16 22:31:57.060120 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:57.060033 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b89fb2b1-14a7-472e-8b1f-5a7a5b761d52-home\") pod \"e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr\" (UID: \"b89fb2b1-14a7-472e-8b1f-5a7a5b761d52\") " pod="ai-tenant-e2e-mt-601103b4/e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr" Jul 16 22:31:57.060120 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:57.060056 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfj7d\" (UniqueName: \"kubernetes.io/projected/b89fb2b1-14a7-472e-8b1f-5a7a5b761d52-kube-api-access-tfj7d\") pod \"e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr\" (UID: \"b89fb2b1-14a7-472e-8b1f-5a7a5b761d52\") " pod="ai-tenant-e2e-mt-601103b4/e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr" Jul 16 22:31:57.060120 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:57.060093 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b89fb2b1-14a7-472e-8b1f-5a7a5b761d52-tmp-dir\") pod \"e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr\" (UID: \"b89fb2b1-14a7-472e-8b1f-5a7a5b761d52\") " pod="ai-tenant-e2e-mt-601103b4/e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr" Jul 16 22:31:57.060375 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:57.060166 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b89fb2b1-14a7-472e-8b1f-5a7a5b761d52-dshm\") pod \"e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr\" (UID: \"b89fb2b1-14a7-472e-8b1f-5a7a5b761d52\") " pod="ai-tenant-e2e-mt-601103b4/e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr" Jul 16 22:31:57.060375 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:57.060327 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b89fb2b1-14a7-472e-8b1f-5a7a5b761d52-model-cache\") pod \"e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr\" (UID: \"b89fb2b1-14a7-472e-8b1f-5a7a5b761d52\") " pod="ai-tenant-e2e-mt-601103b4/e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr" Jul 16 22:31:57.060468 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:57.060375 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b89fb2b1-14a7-472e-8b1f-5a7a5b761d52-home\") pod \"e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr\" (UID: \"b89fb2b1-14a7-472e-8b1f-5a7a5b761d52\") " pod="ai-tenant-e2e-mt-601103b4/e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr" Jul 16 22:31:57.060538 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:57.060522 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b89fb2b1-14a7-472e-8b1f-5a7a5b761d52-tmp-dir\") pod \"e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr\" (UID: \"b89fb2b1-14a7-472e-8b1f-5a7a5b761d52\") " pod="ai-tenant-e2e-mt-601103b4/e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr" Jul 16 22:31:57.062325 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:57.062307 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b89fb2b1-14a7-472e-8b1f-5a7a5b761d52-dshm\") pod \"e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr\" (UID: \"b89fb2b1-14a7-472e-8b1f-5a7a5b761d52\") " pod="ai-tenant-e2e-mt-601103b4/e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr" Jul 16 22:31:57.062484 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:57.062469 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b89fb2b1-14a7-472e-8b1f-5a7a5b761d52-tls-certs\") pod \"e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr\" (UID: \"b89fb2b1-14a7-472e-8b1f-5a7a5b761d52\") " pod="ai-tenant-e2e-mt-601103b4/e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr" Jul 16 22:31:57.073898 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:57.073837 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfj7d\" (UniqueName: \"kubernetes.io/projected/b89fb2b1-14a7-472e-8b1f-5a7a5b761d52-kube-api-access-tfj7d\") pod \"e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr\" (UID: \"b89fb2b1-14a7-472e-8b1f-5a7a5b761d52\") " pod="ai-tenant-e2e-mt-601103b4/e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr" Jul 16 22:31:57.132605 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:57.132578 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-601103b4/e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr" Jul 16 22:31:57.288003 ip-10-0-128-255 kubenswrapper[2585]: W0716 22:31:57.287960 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb89fb2b1_14a7_472e_8b1f_5a7a5b761d52.slice/crio-c1a76108d7aa51101d8e772d0a92a767d6ab8ae339323f53db7777d8bdfff5d1 WatchSource:0}: Error finding container c1a76108d7aa51101d8e772d0a92a767d6ab8ae339323f53db7777d8bdfff5d1: Status 404 returned error can't find the container with id c1a76108d7aa51101d8e772d0a92a767d6ab8ae339323f53db7777d8bdfff5d1 Jul 16 22:31:57.291031 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:57.291010 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-601103b4/e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr"] Jul 16 22:31:58.060884 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:58.060844 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-601103b4/e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr" event={"ID":"b89fb2b1-14a7-472e-8b1f-5a7a5b761d52","Type":"ContainerStarted","Data":"d6795bc9b6cd82e5f153ee8844e193a130aaa1dd7a3e0fdae0237b3fa8a91a4d"} Jul 16 22:31:58.060884 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:58.060890 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-601103b4/e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr" event={"ID":"b89fb2b1-14a7-472e-8b1f-5a7a5b761d52","Type":"ContainerStarted","Data":"c1a76108d7aa51101d8e772d0a92a767d6ab8ae339323f53db7777d8bdfff5d1"} Jul 16 22:31:58.103508 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:31:58.103458 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-601103b4/e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr" podStartSLOduration=2.103443702 podStartE2EDuration="2.103443702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:31:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:31:58.101026626 +0000 UTC m=+2666.723469574" watchObservedRunningTime="2026-07-16 22:31:58.103443702 +0000 UTC m=+2666.725886693" Jul 16 22:32:00.257917 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:32:00.257885 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:32:07.133000 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:32:07.132967 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-601103b4/e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr" Jul 16 22:32:07.133585 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:32:07.133012 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-601103b4/e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr" Jul 16 22:32:07.145754 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:32:07.145724 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-601103b4/e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr" Jul 16 22:32:08.123177 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:32:08.123150 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-601103b4/e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr" Jul 16 22:32:10.493405 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:32:10.493377 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:32:16.329578 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:32:16.329537 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:32:30.080190 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:32:30.080101 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:32:34.312124 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:32:34.312083 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:32:34.590282 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:32:34.590157 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:32:39.554206 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:32:39.554168 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-601103b4/e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr"] Jul 16 22:32:39.677915 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:32:39.677884 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:32:47.355953 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:32:47.355918 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:32:47.456936 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:32:47.456903 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:32:48.844662 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:32:48.844625 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:32:50.456212 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:32:50.456184 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:32:54.574550 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:32:54.574518 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-601103b4/e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr" podUID="b89fb2b1-14a7-472e-8b1f-5a7a5b761d52" containerName="main" containerID="cri-o://d6795bc9b6cd82e5f153ee8844e193a130aaa1dd7a3e0fdae0237b3fa8a91a4d" gracePeriod=45 Jul 16 22:32:54.816082 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:32:54.816060 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-601103b4/e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr" Jul 16 22:32:54.971491 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:32:54.971412 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b89fb2b1-14a7-472e-8b1f-5a7a5b761d52-dshm\") pod \"b89fb2b1-14a7-472e-8b1f-5a7a5b761d52\" (UID: \"b89fb2b1-14a7-472e-8b1f-5a7a5b761d52\") " Jul 16 22:32:54.971491 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:32:54.971452 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b89fb2b1-14a7-472e-8b1f-5a7a5b761d52-model-cache\") pod \"b89fb2b1-14a7-472e-8b1f-5a7a5b761d52\" (UID: \"b89fb2b1-14a7-472e-8b1f-5a7a5b761d52\") " Jul 16 22:32:54.971491 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:32:54.971476 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b89fb2b1-14a7-472e-8b1f-5a7a5b761d52-tls-certs\") pod \"b89fb2b1-14a7-472e-8b1f-5a7a5b761d52\" (UID: \"b89fb2b1-14a7-472e-8b1f-5a7a5b761d52\") " Jul 16 22:32:54.971771 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:32:54.971528 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b89fb2b1-14a7-472e-8b1f-5a7a5b761d52-home\") pod \"b89fb2b1-14a7-472e-8b1f-5a7a5b761d52\" (UID: \"b89fb2b1-14a7-472e-8b1f-5a7a5b761d52\") " Jul 16 22:32:54.971771 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:32:54.971546 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfj7d\" (UniqueName: \"kubernetes.io/projected/b89fb2b1-14a7-472e-8b1f-5a7a5b761d52-kube-api-access-tfj7d\") pod \"b89fb2b1-14a7-472e-8b1f-5a7a5b761d52\" (UID: \"b89fb2b1-14a7-472e-8b1f-5a7a5b761d52\") " Jul 16 22:32:54.971771 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:32:54.971561 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b89fb2b1-14a7-472e-8b1f-5a7a5b761d52-tmp-dir\") pod \"b89fb2b1-14a7-472e-8b1f-5a7a5b761d52\" (UID: \"b89fb2b1-14a7-472e-8b1f-5a7a5b761d52\") " Jul 16 22:32:54.971771 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:32:54.971698 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b89fb2b1-14a7-472e-8b1f-5a7a5b761d52-model-cache" (OuterVolumeSpecName: "model-cache") pod "b89fb2b1-14a7-472e-8b1f-5a7a5b761d52" (UID: "b89fb2b1-14a7-472e-8b1f-5a7a5b761d52"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:32:54.971967 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:32:54.971789 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b89fb2b1-14a7-472e-8b1f-5a7a5b761d52-home" (OuterVolumeSpecName: "home") pod "b89fb2b1-14a7-472e-8b1f-5a7a5b761d52" (UID: "b89fb2b1-14a7-472e-8b1f-5a7a5b761d52"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:32:54.971967 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:32:54.971875 2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b89fb2b1-14a7-472e-8b1f-5a7a5b761d52-model-cache\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:32:54.971967 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:32:54.971894 2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b89fb2b1-14a7-472e-8b1f-5a7a5b761d52-home\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:32:54.971967 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:32:54.971901 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b89fb2b1-14a7-472e-8b1f-5a7a5b761d52-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b89fb2b1-14a7-472e-8b1f-5a7a5b761d52" (UID: "b89fb2b1-14a7-472e-8b1f-5a7a5b761d52"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:32:54.973577 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:32:54.973554 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b89fb2b1-14a7-472e-8b1f-5a7a5b761d52-dshm" (OuterVolumeSpecName: "dshm") pod "b89fb2b1-14a7-472e-8b1f-5a7a5b761d52" (UID: "b89fb2b1-14a7-472e-8b1f-5a7a5b761d52"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:32:54.973676 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:32:54.973624 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b89fb2b1-14a7-472e-8b1f-5a7a5b761d52-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b89fb2b1-14a7-472e-8b1f-5a7a5b761d52" (UID: "b89fb2b1-14a7-472e-8b1f-5a7a5b761d52"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 22:32:54.973747 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:32:54.973727 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b89fb2b1-14a7-472e-8b1f-5a7a5b761d52-kube-api-access-tfj7d" (OuterVolumeSpecName: "kube-api-access-tfj7d") pod "b89fb2b1-14a7-472e-8b1f-5a7a5b761d52" (UID: "b89fb2b1-14a7-472e-8b1f-5a7a5b761d52"). InnerVolumeSpecName "kube-api-access-tfj7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 22:32:55.073290 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:32:55.073256 2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b89fb2b1-14a7-472e-8b1f-5a7a5b761d52-tls-certs\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:32:55.073290 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:32:55.073284 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfj7d\" (UniqueName: \"kubernetes.io/projected/b89fb2b1-14a7-472e-8b1f-5a7a5b761d52-kube-api-access-tfj7d\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:32:55.073499 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:32:55.073298 2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b89fb2b1-14a7-472e-8b1f-5a7a5b761d52-tmp-dir\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:32:55.073499 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:32:55.073314 2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b89fb2b1-14a7-472e-8b1f-5a7a5b761d52-dshm\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:32:55.345484 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:32:55.345455 2585 generic.go:356] "Generic (PLEG): container finished" podID="b89fb2b1-14a7-472e-8b1f-5a7a5b761d52" containerID="d6795bc9b6cd82e5f153ee8844e193a130aaa1dd7a3e0fdae0237b3fa8a91a4d" exitCode=0 Jul 16 22:32:55.345640 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:32:55.345495 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-601103b4/e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr" Jul 16 22:32:55.345640 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:32:55.345530 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-601103b4/e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr" event={"ID":"b89fb2b1-14a7-472e-8b1f-5a7a5b761d52","Type":"ContainerDied","Data":"d6795bc9b6cd82e5f153ee8844e193a130aaa1dd7a3e0fdae0237b3fa8a91a4d"} Jul 16 22:32:55.345640 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:32:55.345570 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-601103b4/e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr" event={"ID":"b89fb2b1-14a7-472e-8b1f-5a7a5b761d52","Type":"ContainerDied","Data":"c1a76108d7aa51101d8e772d0a92a767d6ab8ae339323f53db7777d8bdfff5d1"} Jul 16 22:32:55.345640 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:32:55.345586 2585 scope.go:117] "RemoveContainer" containerID="d6795bc9b6cd82e5f153ee8844e193a130aaa1dd7a3e0fdae0237b3fa8a91a4d" Jul 16 22:32:55.361655 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:32:55.361637 2585 scope.go:117] "RemoveContainer" containerID="d6795bc9b6cd82e5f153ee8844e193a130aaa1dd7a3e0fdae0237b3fa8a91a4d" Jul 16 22:32:55.361898 ip-10-0-128-255 kubenswrapper[2585]: E0716 22:32:55.361878 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6795bc9b6cd82e5f153ee8844e193a130aaa1dd7a3e0fdae0237b3fa8a91a4d\": container with ID starting with d6795bc9b6cd82e5f153ee8844e193a130aaa1dd7a3e0fdae0237b3fa8a91a4d not found: ID does not exist" containerID="d6795bc9b6cd82e5f153ee8844e193a130aaa1dd7a3e0fdae0237b3fa8a91a4d" Jul 16 22:32:55.361964 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:32:55.361907 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6795bc9b6cd82e5f153ee8844e193a130aaa1dd7a3e0fdae0237b3fa8a91a4d"} err="failed to get container status \"d6795bc9b6cd82e5f153ee8844e193a130aaa1dd7a3e0fdae0237b3fa8a91a4d\": rpc error: code = NotFound desc = could not find container \"d6795bc9b6cd82e5f153ee8844e193a130aaa1dd7a3e0fdae0237b3fa8a91a4d\": container with ID starting with d6795bc9b6cd82e5f153ee8844e193a130aaa1dd7a3e0fdae0237b3fa8a91a4d not found: ID does not exist" Jul 16 22:32:55.389510 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:32:55.389454 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-601103b4/e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr"] Jul 16 22:32:55.391613 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:32:55.391594 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-601103b4/e2e-lifecycle-model-601103b4-kserve-d86799fff-snjtr"] Jul 16 22:32:55.935276 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:32:55.935240 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b89fb2b1-14a7-472e-8b1f-5a7a5b761d52" path="/var/lib/kubelet/pods/b89fb2b1-14a7-472e-8b1f-5a7a5b761d52/volumes" Jul 16 22:33:15.925367 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:33:15.925336 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:33:17.241688 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:33:17.241656 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:33:22.969337 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:33:22.969299 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:33:23.075888 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:33:23.075856 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:33:26.804109 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:33:26.804078 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:33:27.033526 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:33:27.033495 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:33:28.048894 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:33:28.048864 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:33:28.150651 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:33:28.150619 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:33:28.257434 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:33:28.257409 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:33:28.965583 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:33:28.965557 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:33:29.097810 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:33:29.097779 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:33:29.194099 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:33:29.194068 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:33:30.522592 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:33:30.522563 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:33:31.228265 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:33:31.228233 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:33:36.839662 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:33:36.839621 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:33:36.940040 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:33:36.940007 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:33:37.509117 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:33:37.509085 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:33:44.162094 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:33:44.162061 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:33:44.278535 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:33:44.278502 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:33:44.386480 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:33:44.386449 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:33:45.301279 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:33:45.301243 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:33:54.888210 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:33:54.888143 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:33:55.246561 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:33:55.246487 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:34:16.183436 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:16.183402 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:34:21.996848 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:21.996813 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:34:22.062836 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:22.062801 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:34:22.342008 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:22.341976 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:34:22.746667 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:22.746588 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:34:23.121181 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:23.121156 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:34:25.612001 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:25.611971 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:34:25.717094 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:25.717066 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:34:25.875316 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:25.875233 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:34:26.033327 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:26.033292 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:34:26.472550 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:26.472512 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:34:26.573640 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:26.573604 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:34:29.995827 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:29.995793 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:34:30.333074 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:30.333046 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:34:30.622302 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:30.622198 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8"] Jul 16 22:34:30.622671 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:30.622654 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="861ea66c-1923-4fde-a011-660b5dbd94b4" containerName="revoke-keys" Jul 16 22:34:30.622732 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:30.622674 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="861ea66c-1923-4fde-a011-660b5dbd94b4" containerName="revoke-keys" Jul 16 22:34:30.622732 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:30.622694 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b89fb2b1-14a7-472e-8b1f-5a7a5b761d52" containerName="main" Jul 16 22:34:30.622732 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:30.622705 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b89fb2b1-14a7-472e-8b1f-5a7a5b761d52" containerName="main" Jul 16 22:34:30.622884 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:30.622801 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b89fb2b1-14a7-472e-8b1f-5a7a5b761d52" containerName="main" Jul 16 22:34:30.622884 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:30.622815 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="861ea66c-1923-4fde-a011-660b5dbd94b4" containerName="revoke-keys" Jul 16 22:34:30.625885 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:30.625868 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" Jul 16 22:34:30.628476 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:30.628455 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-cca4a2-openshift-default-dockercfg-64w5h\"" Jul 16 22:34:30.628551 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:30.628480 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 16 22:34:30.639856 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:30.639829 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf384beb-6e7c-4afe-add4-a195469fee9a-istio-token\") pod \"e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8\" (UID: \"cf384beb-6e7c-4afe-add4-a195469fee9a\") " pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" Jul 16 22:34:30.639983 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:30.639866 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf384beb-6e7c-4afe-add4-a195469fee9a-istio-podinfo\") pod \"e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8\" (UID: \"cf384beb-6e7c-4afe-add4-a195469fee9a\") " pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" Jul 16 22:34:30.639983 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:30.639891 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf384beb-6e7c-4afe-add4-a195469fee9a-credential-socket\") pod \"e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8\" (UID: \"cf384beb-6e7c-4afe-add4-a195469fee9a\") " pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" Jul 16 22:34:30.640092 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:30.639997 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf384beb-6e7c-4afe-add4-a195469fee9a-istio-data\") pod \"e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8\" (UID: \"cf384beb-6e7c-4afe-add4-a195469fee9a\") " pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" Jul 16 22:34:30.640092 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:30.640034 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf384beb-6e7c-4afe-add4-a195469fee9a-istiod-ca-cert\") pod \"e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8\" (UID: \"cf384beb-6e7c-4afe-add4-a195469fee9a\") " pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" Jul 16 22:34:30.640092 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:30.640065 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8"] Jul 16 22:34:30.640293 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:30.640095 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf384beb-6e7c-4afe-add4-a195469fee9a-workload-socket\") pod \"e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8\" (UID: \"cf384beb-6e7c-4afe-add4-a195469fee9a\") " pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" Jul 16 22:34:30.640293 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:30.640130 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddh9x\" (UniqueName: \"kubernetes.io/projected/cf384beb-6e7c-4afe-add4-a195469fee9a-kube-api-access-ddh9x\") pod \"e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8\" (UID: \"cf384beb-6e7c-4afe-add4-a195469fee9a\") " pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" Jul 16 22:34:30.640293 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:30.640153 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf384beb-6e7c-4afe-add4-a195469fee9a-workload-certs\") pod \"e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8\" (UID: \"cf384beb-6e7c-4afe-add4-a195469fee9a\") " pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" Jul 16 22:34:30.640293 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:30.640173 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf384beb-6e7c-4afe-add4-a195469fee9a-istio-envoy\") pod \"e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8\" (UID: \"cf384beb-6e7c-4afe-add4-a195469fee9a\") " pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" Jul 16 22:34:30.662169 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:30.662148 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:34:30.741092 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:30.741053 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf384beb-6e7c-4afe-add4-a195469fee9a-istio-data\") pod \"e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8\" (UID: \"cf384beb-6e7c-4afe-add4-a195469fee9a\") " pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" Jul 16 22:34:30.741092 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:30.741091 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf384beb-6e7c-4afe-add4-a195469fee9a-istiod-ca-cert\") pod \"e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8\" (UID: \"cf384beb-6e7c-4afe-add4-a195469fee9a\") " pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" Jul 16 22:34:30.741361 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:30.741118 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf384beb-6e7c-4afe-add4-a195469fee9a-workload-socket\") pod \"e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8\" (UID: \"cf384beb-6e7c-4afe-add4-a195469fee9a\") " pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" Jul 16 22:34:30.741361 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:30.741155 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddh9x\" (UniqueName: \"kubernetes.io/projected/cf384beb-6e7c-4afe-add4-a195469fee9a-kube-api-access-ddh9x\") pod \"e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8\" (UID: \"cf384beb-6e7c-4afe-add4-a195469fee9a\") " pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" Jul 16 22:34:30.741361 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:30.741188 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf384beb-6e7c-4afe-add4-a195469fee9a-workload-certs\") pod \"e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8\" (UID: \"cf384beb-6e7c-4afe-add4-a195469fee9a\") " pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" Jul 16 22:34:30.741361 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:30.741211 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf384beb-6e7c-4afe-add4-a195469fee9a-istio-envoy\") pod \"e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8\" (UID: \"cf384beb-6e7c-4afe-add4-a195469fee9a\") " pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" Jul 16 22:34:30.741361 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:30.741285 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf384beb-6e7c-4afe-add4-a195469fee9a-istio-token\") pod \"e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8\" (UID: \"cf384beb-6e7c-4afe-add4-a195469fee9a\") " pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" Jul 16 22:34:30.741361 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:30.741330 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf384beb-6e7c-4afe-add4-a195469fee9a-istio-podinfo\") pod \"e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8\" (UID: \"cf384beb-6e7c-4afe-add4-a195469fee9a\") " pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" Jul 16 22:34:30.741701 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:30.741364 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf384beb-6e7c-4afe-add4-a195469fee9a-credential-socket\") pod \"e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8\" (UID: \"cf384beb-6e7c-4afe-add4-a195469fee9a\") " pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" Jul 16 22:34:30.741701 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:30.741522 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf384beb-6e7c-4afe-add4-a195469fee9a-workload-socket\") pod \"e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8\" (UID: \"cf384beb-6e7c-4afe-add4-a195469fee9a\") " pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" Jul 16 22:34:30.741701 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:30.741588 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf384beb-6e7c-4afe-add4-a195469fee9a-workload-certs\") pod \"e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8\" (UID: \"cf384beb-6e7c-4afe-add4-a195469fee9a\") " pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" Jul 16 22:34:30.741701 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:30.741610 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf384beb-6e7c-4afe-add4-a195469fee9a-istio-data\") pod \"e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8\" (UID: \"cf384beb-6e7c-4afe-add4-a195469fee9a\") " pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" Jul 16 22:34:30.741701 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:30.741634 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf384beb-6e7c-4afe-add4-a195469fee9a-credential-socket\") pod \"e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8\" (UID: \"cf384beb-6e7c-4afe-add4-a195469fee9a\") " pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" Jul 16 22:34:30.741981 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:30.741963 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf384beb-6e7c-4afe-add4-a195469fee9a-istiod-ca-cert\") pod \"e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8\" (UID: \"cf384beb-6e7c-4afe-add4-a195469fee9a\") " pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" Jul 16 22:34:30.743534 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:30.743509 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf384beb-6e7c-4afe-add4-a195469fee9a-istio-envoy\") pod \"e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8\" (UID: \"cf384beb-6e7c-4afe-add4-a195469fee9a\") " pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" Jul 16 22:34:30.743783 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:30.743763 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf384beb-6e7c-4afe-add4-a195469fee9a-istio-podinfo\") pod \"e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8\" (UID: \"cf384beb-6e7c-4afe-add4-a195469fee9a\") " pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" Jul 16 22:34:30.751308 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:30.751242 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddh9x\" (UniqueName: \"kubernetes.io/projected/cf384beb-6e7c-4afe-add4-a195469fee9a-kube-api-access-ddh9x\") pod \"e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8\" (UID: \"cf384beb-6e7c-4afe-add4-a195469fee9a\") " pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" Jul 16 22:34:30.751308 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:30.751247 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf384beb-6e7c-4afe-add4-a195469fee9a-istio-token\") pod \"e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8\" (UID: \"cf384beb-6e7c-4afe-add4-a195469fee9a\") " pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" Jul 16 22:34:30.935858 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:30.935782 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" Jul 16 22:34:31.070789 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:31.070767 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8"] Jul 16 22:34:31.072825 ip-10-0-128-255 kubenswrapper[2585]: W0716 22:34:31.072787 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf384beb_6e7c_4afe_add4_a195469fee9a.slice/crio-5ff3ba3ff2025d354a9022277f360408cb3b7349267e35bd1eb58df714b1a39d WatchSource:0}: Error finding container 5ff3ba3ff2025d354a9022277f360408cb3b7349267e35bd1eb58df714b1a39d: Status 404 returned error can't find the container with id 5ff3ba3ff2025d354a9022277f360408cb3b7349267e35bd1eb58df714b1a39d Jul 16 22:34:31.075154 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:31.075135 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 22:34:31.222251 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:31.222094 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:34:31.825923 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:31.825886 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" event={"ID":"cf384beb-6e7c-4afe-add4-a195469fee9a","Type":"ContainerStarted","Data":"5ff3ba3ff2025d354a9022277f360408cb3b7349267e35bd1eb58df714b1a39d"} Jul 16 22:34:33.578438 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:33.578397 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 22:34:33.578751 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:33.578473 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 22:34:33.578751 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:33.578503 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 22:34:33.839612 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:33.839513 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" event={"ID":"cf384beb-6e7c-4afe-add4-a195469fee9a","Type":"ContainerStarted","Data":"dd5a0aa5534daf29fc8d11334ea97fa245f7976b1e98ed035a89592a293ea9ca"} Jul 16 22:34:33.863761 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:33.863715 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" podStartSLOduration=1.36088905 podStartE2EDuration="3.863699698s" totalImagesPullingTime="2.502810648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:34:33.861688191 +0000 UTC m=+2822.484131136" watchObservedRunningTime="2026-07-16 22:34:33.863699698 +0000 UTC m=+2822.486142705" Jul 16 22:34:33.894691 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:33.894663 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:34:33.936279 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:33.936261 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" Jul 16 22:34:34.940878 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:34.940843 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" Jul 16 22:34:35.806822 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:35.806783 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:34:35.850180 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:35.850156 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" Jul 16 22:34:35.851159 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:35.851137 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" Jul 16 22:34:35.896263 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:35.896202 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:34:36.057445 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:36.057348 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:34:36.410428 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:36.410331 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:34:36.597125 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:36.597087 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:34:36.696259 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:36.696159 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:34:40.278926 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:40.278890 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:34:45.119269 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:45.119211 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:34:47.663117 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:47.663078 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:34:47.732018 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:47.731986 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:34:48.919161 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:48.919088 2585 generic.go:356] "Generic (PLEG): container finished" podID="cf384beb-6e7c-4afe-add4-a195469fee9a" containerID="dd5a0aa5534daf29fc8d11334ea97fa245f7976b1e98ed035a89592a293ea9ca" exitCode=0 Jul 16 22:34:48.919549 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:48.919162 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" event={"ID":"cf384beb-6e7c-4afe-add4-a195469fee9a","Type":"ContainerDied","Data":"dd5a0aa5534daf29fc8d11334ea97fa245f7976b1e98ed035a89592a293ea9ca"} Jul 16 22:34:48.919549 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:48.919531 2585 scope.go:117] "RemoveContainer" containerID="dd5a0aa5534daf29fc8d11334ea97fa245f7976b1e98ed035a89592a293ea9ca" Jul 16 22:34:48.921504 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:48.921464 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 22:34:48.921572 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:48.921540 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 22:34:48.921612 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:48.921576 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 22:34:49.925959 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:49.925925 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" event={"ID":"cf384beb-6e7c-4afe-add4-a195469fee9a","Type":"ContainerStarted","Data":"c08d2089163bfc31eab86fae76a1d36a2eda12279167bd0c2ea39a2d472c1e33"} Jul 16 22:34:49.936111 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:49.936084 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" Jul 16 22:34:49.937272 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:49.937248 2585 patch_prober.go:28] interesting pod/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 16 22:34:49.937391 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:49.937296 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" podUID="cf384beb-6e7c-4afe-add4-a195469fee9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 16 22:34:50.850813 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:50.850781 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" Jul 16 22:34:50.936341 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:50.936302 2585 patch_prober.go:28] interesting pod/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 16 22:34:50.936820 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:50.936368 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" podUID="cf384beb-6e7c-4afe-add4-a195469fee9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 16 22:34:51.940164 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:51.940120 2585 patch_prober.go:28] interesting pod/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 16 22:34:51.940693 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:51.940184 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" podUID="cf384beb-6e7c-4afe-add4-a195469fee9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 16 22:34:52.937759 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:52.937713 2585 patch_prober.go:28] interesting pod/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 16 22:34:52.938099 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:52.937789 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" podUID="cf384beb-6e7c-4afe-add4-a195469fee9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 16 22:34:53.937066 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:53.937030 2585 patch_prober.go:28] interesting pod/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 16 22:34:53.937691 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:53.937661 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" podUID="cf384beb-6e7c-4afe-add4-a195469fee9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 16 22:34:54.939781 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:54.939750 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" Jul 16 22:34:54.940644 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:54.940626 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" Jul 16 22:34:57.667685 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:57.667651 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:34:57.771802 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:57.771755 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:34:59.852463 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:59.852429 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-c36d62/test-model-c36d62-kserve-66cdddcb69-q5kvb"] Jul 16 22:34:59.862411 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:59.862393 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c36d62/test-model-c36d62-kserve-66cdddcb69-q5kvb" Jul 16 22:34:59.865364 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:59.865340 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c36d62\"/\"openshift-service-ca.crt\"" Jul 16 22:34:59.865482 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:59.865397 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c36d62\"/\"kube-root-ca.crt\"" Jul 16 22:34:59.866342 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:59.866320 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c36d62\"/\"test-model-c36d62-kserve-self-signed-certs\"" Jul 16 22:34:59.866458 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:59.866367 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c36d62\"/\"default-dockercfg-8znhh\"" Jul 16 22:34:59.875082 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:59.875057 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c36d62/test-model-c36d62-kserve-66cdddcb69-q5kvb"] Jul 16 22:34:59.880487 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:34:59.880468 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:35:00.005090 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:00.005064 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6710203-6b35-4a20-aed9-a9d1e1eaa239-tls-certs\") pod \"test-model-c36d62-kserve-66cdddcb69-q5kvb\" (UID: \"f6710203-6b35-4a20-aed9-a9d1e1eaa239\") " pod="ai-tenant-e2e-inf-a-c36d62/test-model-c36d62-kserve-66cdddcb69-q5kvb" Jul 16 22:35:00.005090 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:00.005093 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6710203-6b35-4a20-aed9-a9d1e1eaa239-dshm\") pod \"test-model-c36d62-kserve-66cdddcb69-q5kvb\" (UID: \"f6710203-6b35-4a20-aed9-a9d1e1eaa239\") " pod="ai-tenant-e2e-inf-a-c36d62/test-model-c36d62-kserve-66cdddcb69-q5kvb" Jul 16 22:35:00.005363 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:00.005117 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6710203-6b35-4a20-aed9-a9d1e1eaa239-home\") pod \"test-model-c36d62-kserve-66cdddcb69-q5kvb\" (UID: \"f6710203-6b35-4a20-aed9-a9d1e1eaa239\") " pod="ai-tenant-e2e-inf-a-c36d62/test-model-c36d62-kserve-66cdddcb69-q5kvb" Jul 16 22:35:00.005363 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:00.005171 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p885f\" (UniqueName: \"kubernetes.io/projected/f6710203-6b35-4a20-aed9-a9d1e1eaa239-kube-api-access-p885f\") pod \"test-model-c36d62-kserve-66cdddcb69-q5kvb\" (UID: \"f6710203-6b35-4a20-aed9-a9d1e1eaa239\") " pod="ai-tenant-e2e-inf-a-c36d62/test-model-c36d62-kserve-66cdddcb69-q5kvb" Jul 16 22:35:00.005363 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:00.005250 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6710203-6b35-4a20-aed9-a9d1e1eaa239-model-cache\") pod \"test-model-c36d62-kserve-66cdddcb69-q5kvb\" (UID: \"f6710203-6b35-4a20-aed9-a9d1e1eaa239\") " pod="ai-tenant-e2e-inf-a-c36d62/test-model-c36d62-kserve-66cdddcb69-q5kvb" Jul 16 22:35:00.005363 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:00.005298 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6710203-6b35-4a20-aed9-a9d1e1eaa239-tmp-dir\") pod \"test-model-c36d62-kserve-66cdddcb69-q5kvb\" (UID: \"f6710203-6b35-4a20-aed9-a9d1e1eaa239\") " pod="ai-tenant-e2e-inf-a-c36d62/test-model-c36d62-kserve-66cdddcb69-q5kvb" Jul 16 22:35:00.106349 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:00.106257 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6710203-6b35-4a20-aed9-a9d1e1eaa239-home\") pod \"test-model-c36d62-kserve-66cdddcb69-q5kvb\" (UID: \"f6710203-6b35-4a20-aed9-a9d1e1eaa239\") " pod="ai-tenant-e2e-inf-a-c36d62/test-model-c36d62-kserve-66cdddcb69-q5kvb" Jul 16 22:35:00.106349 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:00.106319 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p885f\" (UniqueName: \"kubernetes.io/projected/f6710203-6b35-4a20-aed9-a9d1e1eaa239-kube-api-access-p885f\") pod \"test-model-c36d62-kserve-66cdddcb69-q5kvb\" (UID: \"f6710203-6b35-4a20-aed9-a9d1e1eaa239\") " pod="ai-tenant-e2e-inf-a-c36d62/test-model-c36d62-kserve-66cdddcb69-q5kvb" Jul 16 22:35:00.106567 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:00.106348 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6710203-6b35-4a20-aed9-a9d1e1eaa239-model-cache\") pod \"test-model-c36d62-kserve-66cdddcb69-q5kvb\" (UID: \"f6710203-6b35-4a20-aed9-a9d1e1eaa239\") " pod="ai-tenant-e2e-inf-a-c36d62/test-model-c36d62-kserve-66cdddcb69-q5kvb" Jul 16 22:35:00.106567 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:00.106392 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6710203-6b35-4a20-aed9-a9d1e1eaa239-tmp-dir\") pod \"test-model-c36d62-kserve-66cdddcb69-q5kvb\" (UID: \"f6710203-6b35-4a20-aed9-a9d1e1eaa239\") " pod="ai-tenant-e2e-inf-a-c36d62/test-model-c36d62-kserve-66cdddcb69-q5kvb" Jul 16 22:35:00.106567 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:00.106466 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6710203-6b35-4a20-aed9-a9d1e1eaa239-tls-certs\") pod \"test-model-c36d62-kserve-66cdddcb69-q5kvb\" (UID: \"f6710203-6b35-4a20-aed9-a9d1e1eaa239\") " pod="ai-tenant-e2e-inf-a-c36d62/test-model-c36d62-kserve-66cdddcb69-q5kvb" Jul 16 22:35:00.106567 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:00.106490 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6710203-6b35-4a20-aed9-a9d1e1eaa239-dshm\") pod \"test-model-c36d62-kserve-66cdddcb69-q5kvb\" (UID: \"f6710203-6b35-4a20-aed9-a9d1e1eaa239\") " pod="ai-tenant-e2e-inf-a-c36d62/test-model-c36d62-kserve-66cdddcb69-q5kvb" Jul 16 22:35:00.106744 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:00.106683 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6710203-6b35-4a20-aed9-a9d1e1eaa239-home\") pod \"test-model-c36d62-kserve-66cdddcb69-q5kvb\" (UID: \"f6710203-6b35-4a20-aed9-a9d1e1eaa239\") " pod="ai-tenant-e2e-inf-a-c36d62/test-model-c36d62-kserve-66cdddcb69-q5kvb" Jul 16 22:35:00.106744 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:00.106713 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6710203-6b35-4a20-aed9-a9d1e1eaa239-model-cache\") pod \"test-model-c36d62-kserve-66cdddcb69-q5kvb\" (UID: \"f6710203-6b35-4a20-aed9-a9d1e1eaa239\") " pod="ai-tenant-e2e-inf-a-c36d62/test-model-c36d62-kserve-66cdddcb69-q5kvb" Jul 16 22:35:00.106839 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:00.106779 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6710203-6b35-4a20-aed9-a9d1e1eaa239-tmp-dir\") pod \"test-model-c36d62-kserve-66cdddcb69-q5kvb\" (UID: \"f6710203-6b35-4a20-aed9-a9d1e1eaa239\") " pod="ai-tenant-e2e-inf-a-c36d62/test-model-c36d62-kserve-66cdddcb69-q5kvb" Jul 16 22:35:00.108776 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:00.108751 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6710203-6b35-4a20-aed9-a9d1e1eaa239-dshm\") pod \"test-model-c36d62-kserve-66cdddcb69-q5kvb\" (UID: \"f6710203-6b35-4a20-aed9-a9d1e1eaa239\") " pod="ai-tenant-e2e-inf-a-c36d62/test-model-c36d62-kserve-66cdddcb69-q5kvb" Jul 16 22:35:00.108933 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:00.108916 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6710203-6b35-4a20-aed9-a9d1e1eaa239-tls-certs\") pod \"test-model-c36d62-kserve-66cdddcb69-q5kvb\" (UID: \"f6710203-6b35-4a20-aed9-a9d1e1eaa239\") " pod="ai-tenant-e2e-inf-a-c36d62/test-model-c36d62-kserve-66cdddcb69-q5kvb" Jul 16 22:35:00.115087 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:00.115068 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p885f\" (UniqueName: \"kubernetes.io/projected/f6710203-6b35-4a20-aed9-a9d1e1eaa239-kube-api-access-p885f\") pod \"test-model-c36d62-kserve-66cdddcb69-q5kvb\" (UID: \"f6710203-6b35-4a20-aed9-a9d1e1eaa239\") " pod="ai-tenant-e2e-inf-a-c36d62/test-model-c36d62-kserve-66cdddcb69-q5kvb" Jul 16 22:35:00.173128 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:00.173105 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c36d62/test-model-c36d62-kserve-66cdddcb69-q5kvb" Jul 16 22:35:00.315112 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:00.315086 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c36d62/test-model-c36d62-kserve-66cdddcb69-q5kvb"] Jul 16 22:35:00.316298 ip-10-0-128-255 kubenswrapper[2585]: W0716 22:35:00.316269 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6710203_6b35_4a20_aed9_a9d1e1eaa239.slice/crio-4ab7c6c7b6699d58edb154c2bb91ab6d6f394c1a3e8526cccbe776faaec7aac1 WatchSource:0}: Error finding container 4ab7c6c7b6699d58edb154c2bb91ab6d6f394c1a3e8526cccbe776faaec7aac1: Status 404 returned error can't find the container with id 4ab7c6c7b6699d58edb154c2bb91ab6d6f394c1a3e8526cccbe776faaec7aac1 Jul 16 22:35:00.994000 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:00.993965 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c36d62/test-model-c36d62-kserve-66cdddcb69-q5kvb" event={"ID":"f6710203-6b35-4a20-aed9-a9d1e1eaa239","Type":"ContainerStarted","Data":"bbf7ec3d61c7438994b0da2d93c90c4bb7e6582571198850bff99703e523900d"} Jul 16 22:35:00.994427 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:00.994007 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c36d62/test-model-c36d62-kserve-66cdddcb69-q5kvb" event={"ID":"f6710203-6b35-4a20-aed9-a9d1e1eaa239","Type":"ContainerStarted","Data":"4ab7c6c7b6699d58edb154c2bb91ab6d6f394c1a3e8526cccbe776faaec7aac1"} Jul 16 22:35:01.018389 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:01.018339 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-c36d62/test-model-c36d62-kserve-66cdddcb69-q5kvb" podStartSLOduration=2.018323854 podStartE2EDuration="2.018323854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:34:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:35:01.016340201 +0000 UTC m=+2849.638783155" watchObservedRunningTime="2026-07-16 22:35:01.018323854 +0000 UTC m=+2849.640766807" Jul 16 22:35:04.424616 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:04.424579 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:35:10.173457 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:10.173420 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-c36d62/test-model-c36d62-kserve-66cdddcb69-q5kvb" Jul 16 22:35:10.173457 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:10.173467 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c36d62/test-model-c36d62-kserve-66cdddcb69-q5kvb" Jul 16 22:35:10.185808 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:10.185784 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-c36d62/test-model-c36d62-kserve-66cdddcb69-q5kvb" Jul 16 22:35:11.055508 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:11.055477 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c36d62/test-model-c36d62-kserve-66cdddcb69-q5kvb" Jul 16 22:35:17.416433 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:17.416398 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-cca4a2/test-model-cca4a2-kserve-5659cf5765-htxgj"] Jul 16 22:35:17.418856 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:17.418841 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cca4a2/test-model-cca4a2-kserve-5659cf5765-htxgj" Jul 16 22:35:17.422429 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:17.422392 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-cca4a2\"/\"default-dockercfg-jl65h\"" Jul 16 22:35:17.422429 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:17.422398 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-cca4a2\"/\"openshift-service-ca.crt\"" Jul 16 22:35:17.422429 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:17.422428 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-cca4a2\"/\"test-model-cca4a2-kserve-self-signed-certs\"" Jul 16 22:35:17.422429 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:17.422445 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-cca4a2\"/\"kube-root-ca.crt\"" Jul 16 22:35:17.437359 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:17.437337 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-cca4a2/test-model-cca4a2-kserve-5659cf5765-htxgj"] Jul 16 22:35:17.557458 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:17.557430 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5b4daa3-ad40-4f46-a3c0-ba5483ccadab-tls-certs\") pod \"test-model-cca4a2-kserve-5659cf5765-htxgj\" (UID: \"e5b4daa3-ad40-4f46-a3c0-ba5483ccadab\") " pod="ai-tenant-e2e-inf-b-cca4a2/test-model-cca4a2-kserve-5659cf5765-htxgj" Jul 16 22:35:17.557625 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:17.557496 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5b4daa3-ad40-4f46-a3c0-ba5483ccadab-home\") pod \"test-model-cca4a2-kserve-5659cf5765-htxgj\" (UID: \"e5b4daa3-ad40-4f46-a3c0-ba5483ccadab\") " pod="ai-tenant-e2e-inf-b-cca4a2/test-model-cca4a2-kserve-5659cf5765-htxgj" Jul 16 22:35:17.557625 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:17.557581 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5b4daa3-ad40-4f46-a3c0-ba5483ccadab-dshm\") pod \"test-model-cca4a2-kserve-5659cf5765-htxgj\" (UID: \"e5b4daa3-ad40-4f46-a3c0-ba5483ccadab\") " pod="ai-tenant-e2e-inf-b-cca4a2/test-model-cca4a2-kserve-5659cf5765-htxgj" Jul 16 22:35:17.557625 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:17.557618 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5b4daa3-ad40-4f46-a3c0-ba5483ccadab-model-cache\") pod \"test-model-cca4a2-kserve-5659cf5765-htxgj\" (UID: \"e5b4daa3-ad40-4f46-a3c0-ba5483ccadab\") " pod="ai-tenant-e2e-inf-b-cca4a2/test-model-cca4a2-kserve-5659cf5765-htxgj" Jul 16 22:35:17.557786 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:17.557646 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ts4g\" (UniqueName: \"kubernetes.io/projected/e5b4daa3-ad40-4f46-a3c0-ba5483ccadab-kube-api-access-9ts4g\") pod \"test-model-cca4a2-kserve-5659cf5765-htxgj\" (UID: \"e5b4daa3-ad40-4f46-a3c0-ba5483ccadab\") " pod="ai-tenant-e2e-inf-b-cca4a2/test-model-cca4a2-kserve-5659cf5765-htxgj" Jul 16 22:35:17.557786 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:17.557686 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5b4daa3-ad40-4f46-a3c0-ba5483ccadab-tmp-dir\") pod \"test-model-cca4a2-kserve-5659cf5765-htxgj\" (UID: \"e5b4daa3-ad40-4f46-a3c0-ba5483ccadab\") " pod="ai-tenant-e2e-inf-b-cca4a2/test-model-cca4a2-kserve-5659cf5765-htxgj" Jul 16 22:35:17.633546 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:17.633524 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:35:17.658197 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:17.658170 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5b4daa3-ad40-4f46-a3c0-ba5483ccadab-home\") pod \"test-model-cca4a2-kserve-5659cf5765-htxgj\" (UID: \"e5b4daa3-ad40-4f46-a3c0-ba5483ccadab\") " pod="ai-tenant-e2e-inf-b-cca4a2/test-model-cca4a2-kserve-5659cf5765-htxgj" Jul 16 22:35:17.658332 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:17.658261 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5b4daa3-ad40-4f46-a3c0-ba5483ccadab-dshm\") pod \"test-model-cca4a2-kserve-5659cf5765-htxgj\" (UID: \"e5b4daa3-ad40-4f46-a3c0-ba5483ccadab\") " pod="ai-tenant-e2e-inf-b-cca4a2/test-model-cca4a2-kserve-5659cf5765-htxgj" Jul 16 22:35:17.658332 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:17.658309 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5b4daa3-ad40-4f46-a3c0-ba5483ccadab-model-cache\") pod \"test-model-cca4a2-kserve-5659cf5765-htxgj\" (UID: \"e5b4daa3-ad40-4f46-a3c0-ba5483ccadab\") " pod="ai-tenant-e2e-inf-b-cca4a2/test-model-cca4a2-kserve-5659cf5765-htxgj" Jul 16 22:35:17.658425 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:17.658337 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ts4g\" (UniqueName: \"kubernetes.io/projected/e5b4daa3-ad40-4f46-a3c0-ba5483ccadab-kube-api-access-9ts4g\") pod \"test-model-cca4a2-kserve-5659cf5765-htxgj\" (UID: \"e5b4daa3-ad40-4f46-a3c0-ba5483ccadab\") " pod="ai-tenant-e2e-inf-b-cca4a2/test-model-cca4a2-kserve-5659cf5765-htxgj" Jul 16 22:35:17.658477 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:17.658456 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5b4daa3-ad40-4f46-a3c0-ba5483ccadab-tmp-dir\") pod \"test-model-cca4a2-kserve-5659cf5765-htxgj\" (UID: \"e5b4daa3-ad40-4f46-a3c0-ba5483ccadab\") " pod="ai-tenant-e2e-inf-b-cca4a2/test-model-cca4a2-kserve-5659cf5765-htxgj" Jul 16 22:35:17.658539 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:17.658522 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5b4daa3-ad40-4f46-a3c0-ba5483ccadab-tls-certs\") pod \"test-model-cca4a2-kserve-5659cf5765-htxgj\" (UID: \"e5b4daa3-ad40-4f46-a3c0-ba5483ccadab\") " pod="ai-tenant-e2e-inf-b-cca4a2/test-model-cca4a2-kserve-5659cf5765-htxgj" Jul 16 22:35:17.658591 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:17.658550 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5b4daa3-ad40-4f46-a3c0-ba5483ccadab-home\") pod \"test-model-cca4a2-kserve-5659cf5765-htxgj\" (UID: \"e5b4daa3-ad40-4f46-a3c0-ba5483ccadab\") " pod="ai-tenant-e2e-inf-b-cca4a2/test-model-cca4a2-kserve-5659cf5765-htxgj" Jul 16 22:35:17.658671 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:17.658652 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5b4daa3-ad40-4f46-a3c0-ba5483ccadab-model-cache\") pod \"test-model-cca4a2-kserve-5659cf5765-htxgj\" (UID: \"e5b4daa3-ad40-4f46-a3c0-ba5483ccadab\") " pod="ai-tenant-e2e-inf-b-cca4a2/test-model-cca4a2-kserve-5659cf5765-htxgj" Jul 16 22:35:17.658798 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:17.658777 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5b4daa3-ad40-4f46-a3c0-ba5483ccadab-tmp-dir\") pod \"test-model-cca4a2-kserve-5659cf5765-htxgj\" (UID: \"e5b4daa3-ad40-4f46-a3c0-ba5483ccadab\") " pod="ai-tenant-e2e-inf-b-cca4a2/test-model-cca4a2-kserve-5659cf5765-htxgj" Jul 16 22:35:17.660499 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:17.660475 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5b4daa3-ad40-4f46-a3c0-ba5483ccadab-dshm\") pod \"test-model-cca4a2-kserve-5659cf5765-htxgj\" (UID: \"e5b4daa3-ad40-4f46-a3c0-ba5483ccadab\") " pod="ai-tenant-e2e-inf-b-cca4a2/test-model-cca4a2-kserve-5659cf5765-htxgj" Jul 16 22:35:17.660790 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:17.660773 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5b4daa3-ad40-4f46-a3c0-ba5483ccadab-tls-certs\") pod \"test-model-cca4a2-kserve-5659cf5765-htxgj\" (UID: \"e5b4daa3-ad40-4f46-a3c0-ba5483ccadab\") " pod="ai-tenant-e2e-inf-b-cca4a2/test-model-cca4a2-kserve-5659cf5765-htxgj" Jul 16 22:35:17.675924 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:17.675874 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ts4g\" (UniqueName: \"kubernetes.io/projected/e5b4daa3-ad40-4f46-a3c0-ba5483ccadab-kube-api-access-9ts4g\") pod \"test-model-cca4a2-kserve-5659cf5765-htxgj\" (UID: \"e5b4daa3-ad40-4f46-a3c0-ba5483ccadab\") " pod="ai-tenant-e2e-inf-b-cca4a2/test-model-cca4a2-kserve-5659cf5765-htxgj" Jul 16 22:35:17.728532 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:17.728510 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cca4a2/test-model-cca4a2-kserve-5659cf5765-htxgj" Jul 16 22:35:17.871965 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:17.871938 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-cca4a2/test-model-cca4a2-kserve-5659cf5765-htxgj"] Jul 16 22:35:17.873294 ip-10-0-128-255 kubenswrapper[2585]: W0716 22:35:17.873201 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5b4daa3_ad40_4f46_a3c0_ba5483ccadab.slice/crio-a115f41dfa2d9ef85bc9ef5552ba2856334ea62e258424ca7d673a1ba3115e84 WatchSource:0}: Error finding container a115f41dfa2d9ef85bc9ef5552ba2856334ea62e258424ca7d673a1ba3115e84: Status 404 returned error can't find the container with id a115f41dfa2d9ef85bc9ef5552ba2856334ea62e258424ca7d673a1ba3115e84 Jul 16 22:35:18.088301 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:18.088254 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cca4a2/test-model-cca4a2-kserve-5659cf5765-htxgj" event={"ID":"e5b4daa3-ad40-4f46-a3c0-ba5483ccadab","Type":"ContainerStarted","Data":"eb929f81350b6504387e0989b907cdbf6f4d1b33bcfcb94623b0f1dbcde75286"} Jul 16 22:35:18.088301 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:18.088309 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cca4a2/test-model-cca4a2-kserve-5659cf5765-htxgj" event={"ID":"e5b4daa3-ad40-4f46-a3c0-ba5483ccadab","Type":"ContainerStarted","Data":"a115f41dfa2d9ef85bc9ef5552ba2856334ea62e258424ca7d673a1ba3115e84"} Jul 16 22:35:18.116945 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:18.116904 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-cca4a2/test-model-cca4a2-kserve-5659cf5765-htxgj" podStartSLOduration=1.1168916 podStartE2EDuration="1.1168916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:35:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:35:18.115790452 +0000 UTC m=+2866.738233404" watchObservedRunningTime="2026-07-16 22:35:18.1168916 +0000 UTC m=+2866.739334553" Jul 16 22:35:23.916368 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:23.916330 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:35:27.451353 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:27.451318 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:35:27.729679 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:27.729571 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-cca4a2/test-model-cca4a2-kserve-5659cf5765-htxgj" Jul 16 22:35:27.729679 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:27.729632 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-cca4a2/test-model-cca4a2-kserve-5659cf5765-htxgj" Jul 16 22:35:27.742064 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:27.742038 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-cca4a2/test-model-cca4a2-kserve-5659cf5765-htxgj" Jul 16 22:35:28.152538 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:28.152510 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-cca4a2/test-model-cca4a2-kserve-5659cf5765-htxgj" Jul 16 22:35:37.810779 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:37.810745 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:35:44.253277 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:44.253246 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c36d62/test-model-c36d62-kserve-66cdddcb69-q5kvb"] Jul 16 22:35:44.293351 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:44.293318 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:35:45.210369 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:45.210336 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-cca4a2/test-model-cca4a2-kserve-5659cf5765-htxgj"] Jul 16 22:35:45.246916 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:45.246881 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:35:46.091007 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:46.090974 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:35:46.194371 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:46.194343 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:35:50.924525 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:50.924492 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:35:55.886098 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:55.886064 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:35:59.272551 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:59.272495 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-c36d62/test-model-c36d62-kserve-66cdddcb69-q5kvb" podUID="f6710203-6b35-4a20-aed9-a9d1e1eaa239" containerName="main" containerID="cri-o://bbf7ec3d61c7438994b0da2d93c90c4bb7e6582571198850bff99703e523900d" gracePeriod=45 Jul 16 22:35:59.522191 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:59.522167 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c36d62/test-model-c36d62-kserve-66cdddcb69-q5kvb" Jul 16 22:35:59.623134 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:59.623106 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6710203-6b35-4a20-aed9-a9d1e1eaa239-model-cache\") pod \"f6710203-6b35-4a20-aed9-a9d1e1eaa239\" (UID: \"f6710203-6b35-4a20-aed9-a9d1e1eaa239\") " Jul 16 22:35:59.623134 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:59.623138 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6710203-6b35-4a20-aed9-a9d1e1eaa239-tmp-dir\") pod \"f6710203-6b35-4a20-aed9-a9d1e1eaa239\" (UID: \"f6710203-6b35-4a20-aed9-a9d1e1eaa239\") " Jul 16 22:35:59.623383 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:59.623176 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6710203-6b35-4a20-aed9-a9d1e1eaa239-tls-certs\") pod \"f6710203-6b35-4a20-aed9-a9d1e1eaa239\" (UID: \"f6710203-6b35-4a20-aed9-a9d1e1eaa239\") " Jul 16 22:35:59.623383 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:59.623194 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6710203-6b35-4a20-aed9-a9d1e1eaa239-dshm\") pod \"f6710203-6b35-4a20-aed9-a9d1e1eaa239\" (UID: \"f6710203-6b35-4a20-aed9-a9d1e1eaa239\") " Jul 16 22:35:59.623383 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:59.623241 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6710203-6b35-4a20-aed9-a9d1e1eaa239-home\") pod \"f6710203-6b35-4a20-aed9-a9d1e1eaa239\" (UID: \"f6710203-6b35-4a20-aed9-a9d1e1eaa239\") " Jul 16 22:35:59.623383 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:59.623281 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p885f\" (UniqueName: \"kubernetes.io/projected/f6710203-6b35-4a20-aed9-a9d1e1eaa239-kube-api-access-p885f\") pod \"f6710203-6b35-4a20-aed9-a9d1e1eaa239\" (UID: \"f6710203-6b35-4a20-aed9-a9d1e1eaa239\") " Jul 16 22:35:59.623596 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:59.623429 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6710203-6b35-4a20-aed9-a9d1e1eaa239-model-cache" (OuterVolumeSpecName: "model-cache") pod "f6710203-6b35-4a20-aed9-a9d1e1eaa239" (UID: "f6710203-6b35-4a20-aed9-a9d1e1eaa239"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:35:59.623596 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:59.623441 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6710203-6b35-4a20-aed9-a9d1e1eaa239-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f6710203-6b35-4a20-aed9-a9d1e1eaa239" (UID: "f6710203-6b35-4a20-aed9-a9d1e1eaa239"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:35:59.623596 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:59.623532 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6710203-6b35-4a20-aed9-a9d1e1eaa239-home" (OuterVolumeSpecName: "home") pod "f6710203-6b35-4a20-aed9-a9d1e1eaa239" (UID: "f6710203-6b35-4a20-aed9-a9d1e1eaa239"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:35:59.625471 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:59.625447 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6710203-6b35-4a20-aed9-a9d1e1eaa239-dshm" (OuterVolumeSpecName: "dshm") pod "f6710203-6b35-4a20-aed9-a9d1e1eaa239" (UID: "f6710203-6b35-4a20-aed9-a9d1e1eaa239"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:35:59.625578 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:59.625524 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6710203-6b35-4a20-aed9-a9d1e1eaa239-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f6710203-6b35-4a20-aed9-a9d1e1eaa239" (UID: "f6710203-6b35-4a20-aed9-a9d1e1eaa239"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 22:35:59.625578 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:59.625550 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6710203-6b35-4a20-aed9-a9d1e1eaa239-kube-api-access-p885f" (OuterVolumeSpecName: "kube-api-access-p885f") pod "f6710203-6b35-4a20-aed9-a9d1e1eaa239" (UID: "f6710203-6b35-4a20-aed9-a9d1e1eaa239"). InnerVolumeSpecName "kube-api-access-p885f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 22:35:59.724688 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:59.724658 2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6710203-6b35-4a20-aed9-a9d1e1eaa239-home\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:35:59.724688 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:59.724686 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p885f\" (UniqueName: \"kubernetes.io/projected/f6710203-6b35-4a20-aed9-a9d1e1eaa239-kube-api-access-p885f\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:35:59.724885 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:59.724696 2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6710203-6b35-4a20-aed9-a9d1e1eaa239-model-cache\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:35:59.724885 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:59.724715 2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6710203-6b35-4a20-aed9-a9d1e1eaa239-tmp-dir\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:35:59.724885 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:59.724723 2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6710203-6b35-4a20-aed9-a9d1e1eaa239-tls-certs\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:35:59.724885 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:35:59.724730 2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6710203-6b35-4a20-aed9-a9d1e1eaa239-dshm\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:36:00.230502 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:00.230397 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-cca4a2/test-model-cca4a2-kserve-5659cf5765-htxgj" podUID="e5b4daa3-ad40-4f46-a3c0-ba5483ccadab" containerName="main" containerID="cri-o://eb929f81350b6504387e0989b907cdbf6f4d1b33bcfcb94623b0f1dbcde75286" gracePeriod=45 Jul 16 22:36:00.305094 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:00.305061 2585 generic.go:356] "Generic (PLEG): container finished" podID="f6710203-6b35-4a20-aed9-a9d1e1eaa239" containerID="bbf7ec3d61c7438994b0da2d93c90c4bb7e6582571198850bff99703e523900d" exitCode=0 Jul 16 22:36:00.305519 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:00.305116 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c36d62/test-model-c36d62-kserve-66cdddcb69-q5kvb" Jul 16 22:36:00.305519 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:00.305153 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c36d62/test-model-c36d62-kserve-66cdddcb69-q5kvb" event={"ID":"f6710203-6b35-4a20-aed9-a9d1e1eaa239","Type":"ContainerDied","Data":"bbf7ec3d61c7438994b0da2d93c90c4bb7e6582571198850bff99703e523900d"} Jul 16 22:36:00.305519 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:00.305207 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c36d62/test-model-c36d62-kserve-66cdddcb69-q5kvb" event={"ID":"f6710203-6b35-4a20-aed9-a9d1e1eaa239","Type":"ContainerDied","Data":"4ab7c6c7b6699d58edb154c2bb91ab6d6f394c1a3e8526cccbe776faaec7aac1"} Jul 16 22:36:00.305519 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:00.305252 2585 scope.go:117] "RemoveContainer" containerID="bbf7ec3d61c7438994b0da2d93c90c4bb7e6582571198850bff99703e523900d" Jul 16 22:36:00.321436 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:00.321411 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c36d62/test-model-c36d62-kserve-66cdddcb69-q5kvb"] Jul 16 22:36:00.325282 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:00.325261 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-c36d62/test-model-c36d62-kserve-66cdddcb69-q5kvb"] Jul 16 22:36:00.341031 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:00.341009 2585 scope.go:117] "RemoveContainer" containerID="bbf7ec3d61c7438994b0da2d93c90c4bb7e6582571198850bff99703e523900d" Jul 16 22:36:00.341464 ip-10-0-128-255 kubenswrapper[2585]: E0716 22:36:00.341438 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbf7ec3d61c7438994b0da2d93c90c4bb7e6582571198850bff99703e523900d\": container with ID starting with bbf7ec3d61c7438994b0da2d93c90c4bb7e6582571198850bff99703e523900d not found: ID does not exist" containerID="bbf7ec3d61c7438994b0da2d93c90c4bb7e6582571198850bff99703e523900d" Jul 16 22:36:00.341527 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:00.341477 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbf7ec3d61c7438994b0da2d93c90c4bb7e6582571198850bff99703e523900d"} err="failed to get container status \"bbf7ec3d61c7438994b0da2d93c90c4bb7e6582571198850bff99703e523900d\": rpc error: code = NotFound desc = could not find container \"bbf7ec3d61c7438994b0da2d93c90c4bb7e6582571198850bff99703e523900d\": container with ID starting with bbf7ec3d61c7438994b0da2d93c90c4bb7e6582571198850bff99703e523900d not found: ID does not exist" Jul 16 22:36:00.432966 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:00.432932 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:36:00.477373 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:00.477351 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cca4a2/test-model-cca4a2-kserve-5659cf5765-htxgj" Jul 16 22:36:00.632884 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:00.632851 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5b4daa3-ad40-4f46-a3c0-ba5483ccadab-dshm\") pod \"e5b4daa3-ad40-4f46-a3c0-ba5483ccadab\" (UID: \"e5b4daa3-ad40-4f46-a3c0-ba5483ccadab\") " Jul 16 22:36:00.633072 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:00.632890 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5b4daa3-ad40-4f46-a3c0-ba5483ccadab-model-cache\") pod \"e5b4daa3-ad40-4f46-a3c0-ba5483ccadab\" (UID: \"e5b4daa3-ad40-4f46-a3c0-ba5483ccadab\") " Jul 16 22:36:00.633072 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:00.632927 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5b4daa3-ad40-4f46-a3c0-ba5483ccadab-tmp-dir\") pod \"e5b4daa3-ad40-4f46-a3c0-ba5483ccadab\" (UID: \"e5b4daa3-ad40-4f46-a3c0-ba5483ccadab\") " Jul 16 22:36:00.633072 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:00.632953 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5b4daa3-ad40-4f46-a3c0-ba5483ccadab-tls-certs\") pod \"e5b4daa3-ad40-4f46-a3c0-ba5483ccadab\" (UID: \"e5b4daa3-ad40-4f46-a3c0-ba5483ccadab\") " Jul 16 22:36:00.633072 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:00.632977 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ts4g\" (UniqueName: \"kubernetes.io/projected/e5b4daa3-ad40-4f46-a3c0-ba5483ccadab-kube-api-access-9ts4g\") pod \"e5b4daa3-ad40-4f46-a3c0-ba5483ccadab\" (UID: \"e5b4daa3-ad40-4f46-a3c0-ba5483ccadab\") " Jul 16 22:36:00.633072 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:00.633013 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5b4daa3-ad40-4f46-a3c0-ba5483ccadab-home\") pod \"e5b4daa3-ad40-4f46-a3c0-ba5483ccadab\" (UID: \"e5b4daa3-ad40-4f46-a3c0-ba5483ccadab\") " Jul 16 22:36:00.633366 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:00.633144 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5b4daa3-ad40-4f46-a3c0-ba5483ccadab-model-cache" (OuterVolumeSpecName: "model-cache") pod "e5b4daa3-ad40-4f46-a3c0-ba5483ccadab" (UID: "e5b4daa3-ad40-4f46-a3c0-ba5483ccadab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:36:00.633366 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:00.633212 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5b4daa3-ad40-4f46-a3c0-ba5483ccadab-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e5b4daa3-ad40-4f46-a3c0-ba5483ccadab" (UID: "e5b4daa3-ad40-4f46-a3c0-ba5483ccadab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:36:00.633366 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:00.633330 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5b4daa3-ad40-4f46-a3c0-ba5483ccadab-home" (OuterVolumeSpecName: "home") pod "e5b4daa3-ad40-4f46-a3c0-ba5483ccadab" (UID: "e5b4daa3-ad40-4f46-a3c0-ba5483ccadab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:36:00.633514 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:00.633480 2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5b4daa3-ad40-4f46-a3c0-ba5483ccadab-model-cache\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:36:00.633514 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:00.633501 2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5b4daa3-ad40-4f46-a3c0-ba5483ccadab-tmp-dir\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:36:00.633592 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:00.633513 2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5b4daa3-ad40-4f46-a3c0-ba5483ccadab-home\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:36:00.635152 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:00.635130 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5b4daa3-ad40-4f46-a3c0-ba5483ccadab-dshm" (OuterVolumeSpecName: "dshm") pod "e5b4daa3-ad40-4f46-a3c0-ba5483ccadab" (UID: "e5b4daa3-ad40-4f46-a3c0-ba5483ccadab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:36:00.635255 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:00.635133 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5b4daa3-ad40-4f46-a3c0-ba5483ccadab-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e5b4daa3-ad40-4f46-a3c0-ba5483ccadab" (UID: "e5b4daa3-ad40-4f46-a3c0-ba5483ccadab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 22:36:00.635316 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:00.635253 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5b4daa3-ad40-4f46-a3c0-ba5483ccadab-kube-api-access-9ts4g" (OuterVolumeSpecName: "kube-api-access-9ts4g") pod "e5b4daa3-ad40-4f46-a3c0-ba5483ccadab" (UID: "e5b4daa3-ad40-4f46-a3c0-ba5483ccadab"). InnerVolumeSpecName "kube-api-access-9ts4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 22:36:00.734520 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:00.734484 2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5b4daa3-ad40-4f46-a3c0-ba5483ccadab-dshm\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:36:00.734520 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:00.734516 2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5b4daa3-ad40-4f46-a3c0-ba5483ccadab-tls-certs\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:36:00.734520 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:00.734526 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ts4g\" (UniqueName: \"kubernetes.io/projected/e5b4daa3-ad40-4f46-a3c0-ba5483ccadab-kube-api-access-9ts4g\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:36:01.316561 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:01.316528 2585 generic.go:356] "Generic (PLEG): container finished" podID="e5b4daa3-ad40-4f46-a3c0-ba5483ccadab" containerID="eb929f81350b6504387e0989b907cdbf6f4d1b33bcfcb94623b0f1dbcde75286" exitCode=0 Jul 16 22:36:01.317042 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:01.316571 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cca4a2/test-model-cca4a2-kserve-5659cf5765-htxgj" Jul 16 22:36:01.317042 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:01.316590 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cca4a2/test-model-cca4a2-kserve-5659cf5765-htxgj" event={"ID":"e5b4daa3-ad40-4f46-a3c0-ba5483ccadab","Type":"ContainerDied","Data":"eb929f81350b6504387e0989b907cdbf6f4d1b33bcfcb94623b0f1dbcde75286"} Jul 16 22:36:01.317042 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:01.316619 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cca4a2/test-model-cca4a2-kserve-5659cf5765-htxgj" event={"ID":"e5b4daa3-ad40-4f46-a3c0-ba5483ccadab","Type":"ContainerDied","Data":"a115f41dfa2d9ef85bc9ef5552ba2856334ea62e258424ca7d673a1ba3115e84"} Jul 16 22:36:01.317042 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:01.316642 2585 scope.go:117] "RemoveContainer" containerID="eb929f81350b6504387e0989b907cdbf6f4d1b33bcfcb94623b0f1dbcde75286" Jul 16 22:36:01.325623 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:01.325605 2585 scope.go:117] "RemoveContainer" containerID="eb929f81350b6504387e0989b907cdbf6f4d1b33bcfcb94623b0f1dbcde75286" Jul 16 22:36:01.325873 ip-10-0-128-255 kubenswrapper[2585]: E0716 22:36:01.325856 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb929f81350b6504387e0989b907cdbf6f4d1b33bcfcb94623b0f1dbcde75286\": container with ID starting with eb929f81350b6504387e0989b907cdbf6f4d1b33bcfcb94623b0f1dbcde75286 not found: ID does not exist" containerID="eb929f81350b6504387e0989b907cdbf6f4d1b33bcfcb94623b0f1dbcde75286" Jul 16 22:36:01.325912 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:01.325881 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb929f81350b6504387e0989b907cdbf6f4d1b33bcfcb94623b0f1dbcde75286"} err="failed to get container status \"eb929f81350b6504387e0989b907cdbf6f4d1b33bcfcb94623b0f1dbcde75286\": rpc error: code = NotFound desc = could not find container \"eb929f81350b6504387e0989b907cdbf6f4d1b33bcfcb94623b0f1dbcde75286\": container with ID starting with eb929f81350b6504387e0989b907cdbf6f4d1b33bcfcb94623b0f1dbcde75286 not found: ID does not exist" Jul 16 22:36:01.338164 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:01.338143 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-cca4a2/test-model-cca4a2-kserve-5659cf5765-htxgj"] Jul 16 22:36:01.342643 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:01.342622 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-cca4a2/test-model-cca4a2-kserve-5659cf5765-htxgj"] Jul 16 22:36:01.935406 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:01.935367 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5b4daa3-ad40-4f46-a3c0-ba5483ccadab" path="/var/lib/kubelet/pods/e5b4daa3-ad40-4f46-a3c0-ba5483ccadab/volumes" Jul 16 22:36:01.935854 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:01.935833 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6710203-6b35-4a20-aed9-a9d1e1eaa239" path="/var/lib/kubelet/pods/f6710203-6b35-4a20-aed9-a9d1e1eaa239/volumes" Jul 16 22:36:02.108844 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:02.108811 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:36:06.525058 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:06.525022 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:36:06.625910 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:06.625876 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:36:06.745846 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:06.745808 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:36:06.880868 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:06.880838 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:36:11.141465 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:11.141434 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:36:11.346464 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:11.346430 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:36:12.078870 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:12.078839 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:36:12.178916 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:12.178875 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:36:21.924544 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:21.924508 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:36:25.010017 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:25.009978 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:36:26.557864 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:26.557832 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:36:32.583559 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:32.583524 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:36:32.687504 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:32.687473 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:36:37.270962 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:37.270931 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8"] Jul 16 22:36:37.271409 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:37.271130 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" podUID="cf384beb-6e7c-4afe-add4-a195469fee9a" containerName="istio-proxy" containerID="cri-o://c08d2089163bfc31eab86fae76a1d36a2eda12279167bd0c2ea39a2d472c1e33" gracePeriod=30 Jul 16 22:36:37.299465 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:37.299431 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:36:37.536430 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:37.536349 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:36:37.883837 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:37.883806 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:36:38.263837 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:38.263752 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:36:38.281095 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:38.281063 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd"] Jul 16 22:36:38.281452 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:38.281438 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6710203-6b35-4a20-aed9-a9d1e1eaa239" containerName="main" Jul 16 22:36:38.281452 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:38.281453 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6710203-6b35-4a20-aed9-a9d1e1eaa239" containerName="main" Jul 16 22:36:38.281553 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:38.281462 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5b4daa3-ad40-4f46-a3c0-ba5483ccadab" containerName="main" Jul 16 22:36:38.281553 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:38.281467 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5b4daa3-ad40-4f46-a3c0-ba5483ccadab" containerName="main" Jul 16 22:36:38.281553 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:38.281521 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6710203-6b35-4a20-aed9-a9d1e1eaa239" containerName="main" Jul 16 22:36:38.281553 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:38.281531 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5b4daa3-ad40-4f46-a3c0-ba5483ccadab" containerName="main" Jul 16 22:36:38.285820 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:38.285804 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" Jul 16 22:36:38.295008 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:38.294987 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-8247b5-openshift-default-dockercfg-lr9lq\"" Jul 16 22:36:38.329397 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:38.329366 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd"] Jul 16 22:36:38.357993 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:38.357965 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a917555a-1827-4bf1-9958-b76e9ae915ef-credential-socket\") pod \"e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd\" (UID: \"a917555a-1827-4bf1-9958-b76e9ae915ef\") " pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" Jul 16 22:36:38.358139 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:38.357997 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a917555a-1827-4bf1-9958-b76e9ae915ef-istio-data\") pod \"e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd\" (UID: \"a917555a-1827-4bf1-9958-b76e9ae915ef\") " pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" Jul 16 22:36:38.358139 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:38.358016 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a917555a-1827-4bf1-9958-b76e9ae915ef-istio-envoy\") pod \"e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd\" (UID: \"a917555a-1827-4bf1-9958-b76e9ae915ef\") " pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" Jul 16 22:36:38.358139 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:38.358034 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a917555a-1827-4bf1-9958-b76e9ae915ef-workload-certs\") pod \"e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd\" (UID: \"a917555a-1827-4bf1-9958-b76e9ae915ef\") " pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" Jul 16 22:36:38.358139 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:38.358102 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a917555a-1827-4bf1-9958-b76e9ae915ef-workload-socket\") pod \"e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd\" (UID: \"a917555a-1827-4bf1-9958-b76e9ae915ef\") " pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" Jul 16 22:36:38.358391 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:38.358140 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj4l7\" (UniqueName: \"kubernetes.io/projected/a917555a-1827-4bf1-9958-b76e9ae915ef-kube-api-access-pj4l7\") pod \"e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd\" (UID: \"a917555a-1827-4bf1-9958-b76e9ae915ef\") " pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" Jul 16 22:36:38.358391 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:38.358205 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a917555a-1827-4bf1-9958-b76e9ae915ef-istio-token\") pod \"e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd\" (UID: \"a917555a-1827-4bf1-9958-b76e9ae915ef\") " pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" Jul 16 22:36:38.358391 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:38.358269 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a917555a-1827-4bf1-9958-b76e9ae915ef-istiod-ca-cert\") pod \"e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd\" (UID: \"a917555a-1827-4bf1-9958-b76e9ae915ef\") " pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" Jul 16 22:36:38.358391 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:38.358313 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a917555a-1827-4bf1-9958-b76e9ae915ef-istio-podinfo\") pod \"e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd\" (UID: \"a917555a-1827-4bf1-9958-b76e9ae915ef\") " pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" Jul 16 22:36:38.459715 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:38.459683 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a917555a-1827-4bf1-9958-b76e9ae915ef-istio-token\") pod \"e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd\" (UID: \"a917555a-1827-4bf1-9958-b76e9ae915ef\") " pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" Jul 16 22:36:38.459715 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:38.459727 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a917555a-1827-4bf1-9958-b76e9ae915ef-istiod-ca-cert\") pod \"e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd\" (UID: \"a917555a-1827-4bf1-9958-b76e9ae915ef\") " pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" Jul 16 22:36:38.459943 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:38.459756 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a917555a-1827-4bf1-9958-b76e9ae915ef-istio-podinfo\") pod \"e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd\" (UID: \"a917555a-1827-4bf1-9958-b76e9ae915ef\") " pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" Jul 16 22:36:38.459943 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:38.459819 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a917555a-1827-4bf1-9958-b76e9ae915ef-credential-socket\") pod \"e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd\" (UID: \"a917555a-1827-4bf1-9958-b76e9ae915ef\") " pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" Jul 16 22:36:38.459943 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:38.459840 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a917555a-1827-4bf1-9958-b76e9ae915ef-istio-data\") pod \"e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd\" (UID: \"a917555a-1827-4bf1-9958-b76e9ae915ef\") " pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" Jul 16 22:36:38.459943 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:38.459866 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a917555a-1827-4bf1-9958-b76e9ae915ef-istio-envoy\") pod \"e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd\" (UID: \"a917555a-1827-4bf1-9958-b76e9ae915ef\") " pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" Jul 16 22:36:38.459943 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:38.459898 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a917555a-1827-4bf1-9958-b76e9ae915ef-workload-certs\") pod \"e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd\" (UID: \"a917555a-1827-4bf1-9958-b76e9ae915ef\") " pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" Jul 16 22:36:38.459943 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:38.459921 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a917555a-1827-4bf1-9958-b76e9ae915ef-workload-socket\") pod \"e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd\" (UID: \"a917555a-1827-4bf1-9958-b76e9ae915ef\") " pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" Jul 16 22:36:38.459943 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:38.459944 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj4l7\" (UniqueName: \"kubernetes.io/projected/a917555a-1827-4bf1-9958-b76e9ae915ef-kube-api-access-pj4l7\") pod \"e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd\" (UID: \"a917555a-1827-4bf1-9958-b76e9ae915ef\") " pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" Jul 16 22:36:38.460371 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:38.460345 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a917555a-1827-4bf1-9958-b76e9ae915ef-credential-socket\") pod \"e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd\" (UID: \"a917555a-1827-4bf1-9958-b76e9ae915ef\") " pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" Jul 16 22:36:38.460435 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:38.460375 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a917555a-1827-4bf1-9958-b76e9ae915ef-workload-socket\") pod \"e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd\" (UID: \"a917555a-1827-4bf1-9958-b76e9ae915ef\") " pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" Jul 16 22:36:38.460629 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:38.460603 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a917555a-1827-4bf1-9958-b76e9ae915ef-workload-certs\") pod \"e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd\" (UID: \"a917555a-1827-4bf1-9958-b76e9ae915ef\") " pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" Jul 16 22:36:38.460629 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:38.460620 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a917555a-1827-4bf1-9958-b76e9ae915ef-istiod-ca-cert\") pod \"e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd\" (UID: \"a917555a-1827-4bf1-9958-b76e9ae915ef\") " pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" Jul 16 22:36:38.460827 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:38.460637 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a917555a-1827-4bf1-9958-b76e9ae915ef-istio-data\") pod \"e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd\" (UID: \"a917555a-1827-4bf1-9958-b76e9ae915ef\") " pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" Jul 16 22:36:38.462371 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:38.462348 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a917555a-1827-4bf1-9958-b76e9ae915ef-istio-envoy\") pod \"e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd\" (UID: \"a917555a-1827-4bf1-9958-b76e9ae915ef\") " pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" Jul 16 22:36:38.462465 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:38.462451 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a917555a-1827-4bf1-9958-b76e9ae915ef-istio-podinfo\") pod \"e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd\" (UID: \"a917555a-1827-4bf1-9958-b76e9ae915ef\") " pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" Jul 16 22:36:38.471156 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:38.471135 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj4l7\" (UniqueName: \"kubernetes.io/projected/a917555a-1827-4bf1-9958-b76e9ae915ef-kube-api-access-pj4l7\") pod \"e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd\" (UID: \"a917555a-1827-4bf1-9958-b76e9ae915ef\") " pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" Jul 16 22:36:38.473017 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:38.472999 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a917555a-1827-4bf1-9958-b76e9ae915ef-istio-token\") pod \"e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd\" (UID: \"a917555a-1827-4bf1-9958-b76e9ae915ef\") " pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" Jul 16 22:36:38.596243 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:38.596199 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" Jul 16 22:36:38.776684 ip-10-0-128-255 kubenswrapper[2585]: W0716 22:36:38.776640 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda917555a_1827_4bf1_9958_b76e9ae915ef.slice/crio-94be9b291149507bde91e064e054589d9d686b72431b3e4b6c1a5d769affb7ac WatchSource:0}: Error finding container 94be9b291149507bde91e064e054589d9d686b72431b3e4b6c1a5d769affb7ac: Status 404 returned error can't find the container with id 94be9b291149507bde91e064e054589d9d686b72431b3e4b6c1a5d769affb7ac Jul 16 22:36:38.776988 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:38.776970 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd"] Jul 16 22:36:38.778921 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:38.778888 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 22:36:38.779008 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:38.778954 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 22:36:38.779008 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:38.778982 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 22:36:38.832044 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:38.832020 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:36:39.516169 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:39.516136 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" event={"ID":"a917555a-1827-4bf1-9958-b76e9ae915ef","Type":"ContainerStarted","Data":"5e5fe933f972cbbdfc2f7dff12b75acb10a6116c1b43f8723648985addfc867c"} Jul 16 22:36:39.516582 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:39.516189 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" event={"ID":"a917555a-1827-4bf1-9958-b76e9ae915ef","Type":"ContainerStarted","Data":"94be9b291149507bde91e064e054589d9d686b72431b3e4b6c1a5d769affb7ac"} Jul 16 22:36:39.556382 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:39.556338 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" podStartSLOduration=1.55632357 podStartE2EDuration="1.55632357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:36:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:36:39.552918355 +0000 UTC m=+2948.175361308" watchObservedRunningTime="2026-07-16 22:36:39.55632357 +0000 UTC m=+2948.178766595" Jul 16 22:36:39.597183 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:39.597157 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" Jul 16 22:36:39.601905 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:39.601879 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" Jul 16 22:36:39.942191 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:39.942162 2585 patch_prober.go:28] interesting pod/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 16 22:36:39.942394 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:39.942210 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" podUID="cf384beb-6e7c-4afe-add4-a195469fee9a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:36:40.520833 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:40.520803 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" Jul 16 22:36:40.521629 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:40.521611 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" Jul 16 22:36:41.613596 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:41.613561 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:36:41.717816 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:41.717783 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:36:41.935045 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:41.934963 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:36:42.088700 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:42.088666 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:36:42.525204 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:42.525181 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" Jul 16 22:36:42.531977 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:42.531912 2585 generic.go:356] "Generic (PLEG): container finished" podID="cf384beb-6e7c-4afe-add4-a195469fee9a" containerID="c08d2089163bfc31eab86fae76a1d36a2eda12279167bd0c2ea39a2d472c1e33" exitCode=0 Jul 16 22:36:42.532113 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:42.531972 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" Jul 16 22:36:42.532113 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:42.531977 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" event={"ID":"cf384beb-6e7c-4afe-add4-a195469fee9a","Type":"ContainerDied","Data":"c08d2089163bfc31eab86fae76a1d36a2eda12279167bd0c2ea39a2d472c1e33"} Jul 16 22:36:42.532113 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:42.532032 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8" event={"ID":"cf384beb-6e7c-4afe-add4-a195469fee9a","Type":"ContainerDied","Data":"5ff3ba3ff2025d354a9022277f360408cb3b7349267e35bd1eb58df714b1a39d"} Jul 16 22:36:42.532113 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:42.532058 2585 scope.go:117] "RemoveContainer" containerID="c08d2089163bfc31eab86fae76a1d36a2eda12279167bd0c2ea39a2d472c1e33" Jul 16 22:36:42.540581 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:42.540565 2585 scope.go:117] "RemoveContainer" containerID="dd5a0aa5534daf29fc8d11334ea97fa245f7976b1e98ed035a89592a293ea9ca" Jul 16 22:36:42.548937 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:42.548922 2585 scope.go:117] "RemoveContainer" containerID="c08d2089163bfc31eab86fae76a1d36a2eda12279167bd0c2ea39a2d472c1e33" Jul 16 22:36:42.549180 ip-10-0-128-255 kubenswrapper[2585]: E0716 22:36:42.549160 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c08d2089163bfc31eab86fae76a1d36a2eda12279167bd0c2ea39a2d472c1e33\": container with ID starting with c08d2089163bfc31eab86fae76a1d36a2eda12279167bd0c2ea39a2d472c1e33 not found: ID does not exist" containerID="c08d2089163bfc31eab86fae76a1d36a2eda12279167bd0c2ea39a2d472c1e33" Jul 16 22:36:42.549270 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:42.549193 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c08d2089163bfc31eab86fae76a1d36a2eda12279167bd0c2ea39a2d472c1e33"} err="failed to get container status \"c08d2089163bfc31eab86fae76a1d36a2eda12279167bd0c2ea39a2d472c1e33\": rpc error: code = NotFound desc = could not find container \"c08d2089163bfc31eab86fae76a1d36a2eda12279167bd0c2ea39a2d472c1e33\": container with ID starting with c08d2089163bfc31eab86fae76a1d36a2eda12279167bd0c2ea39a2d472c1e33 not found: ID does not exist" Jul 16 22:36:42.549270 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:42.549234 2585 scope.go:117] "RemoveContainer" containerID="dd5a0aa5534daf29fc8d11334ea97fa245f7976b1e98ed035a89592a293ea9ca" Jul 16 22:36:42.549470 ip-10-0-128-255 kubenswrapper[2585]: E0716 22:36:42.549454 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd5a0aa5534daf29fc8d11334ea97fa245f7976b1e98ed035a89592a293ea9ca\": container with ID starting with dd5a0aa5534daf29fc8d11334ea97fa245f7976b1e98ed035a89592a293ea9ca not found: ID does not exist" containerID="dd5a0aa5534daf29fc8d11334ea97fa245f7976b1e98ed035a89592a293ea9ca" Jul 16 22:36:42.549525 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:42.549479 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd5a0aa5534daf29fc8d11334ea97fa245f7976b1e98ed035a89592a293ea9ca"} err="failed to get container status \"dd5a0aa5534daf29fc8d11334ea97fa245f7976b1e98ed035a89592a293ea9ca\": rpc error: code = NotFound desc = could not find container \"dd5a0aa5534daf29fc8d11334ea97fa245f7976b1e98ed035a89592a293ea9ca\": container with ID starting with dd5a0aa5534daf29fc8d11334ea97fa245f7976b1e98ed035a89592a293ea9ca not found: ID does not exist" Jul 16 22:36:42.595495 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:42.595466 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf384beb-6e7c-4afe-add4-a195469fee9a-istio-podinfo\") pod \"cf384beb-6e7c-4afe-add4-a195469fee9a\" (UID: \"cf384beb-6e7c-4afe-add4-a195469fee9a\") " Jul 16 22:36:42.595662 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:42.595524 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf384beb-6e7c-4afe-add4-a195469fee9a-istio-data\") pod \"cf384beb-6e7c-4afe-add4-a195469fee9a\" (UID: \"cf384beb-6e7c-4afe-add4-a195469fee9a\") " Jul 16 22:36:42.595662 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:42.595540 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf384beb-6e7c-4afe-add4-a195469fee9a-workload-socket\") pod \"cf384beb-6e7c-4afe-add4-a195469fee9a\" (UID: \"cf384beb-6e7c-4afe-add4-a195469fee9a\") " Jul 16 22:36:42.595780 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:42.595711 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddh9x\" (UniqueName: \"kubernetes.io/projected/cf384beb-6e7c-4afe-add4-a195469fee9a-kube-api-access-ddh9x\") pod \"cf384beb-6e7c-4afe-add4-a195469fee9a\" (UID: \"cf384beb-6e7c-4afe-add4-a195469fee9a\") " Jul 16 22:36:42.595835 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:42.595780 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf384beb-6e7c-4afe-add4-a195469fee9a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cf384beb-6e7c-4afe-add4-a195469fee9a" (UID: "cf384beb-6e7c-4afe-add4-a195469fee9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:36:42.595835 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:42.595814 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf384beb-6e7c-4afe-add4-a195469fee9a-istio-envoy\") pod \"cf384beb-6e7c-4afe-add4-a195469fee9a\" (UID: \"cf384beb-6e7c-4afe-add4-a195469fee9a\") " Jul 16 22:36:42.595946 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:42.595850 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf384beb-6e7c-4afe-add4-a195469fee9a-credential-socket\") pod \"cf384beb-6e7c-4afe-add4-a195469fee9a\" (UID: \"cf384beb-6e7c-4afe-add4-a195469fee9a\") " Jul 16 22:36:42.595946 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:42.595888 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf384beb-6e7c-4afe-add4-a195469fee9a-istiod-ca-cert\") pod \"cf384beb-6e7c-4afe-add4-a195469fee9a\" (UID: \"cf384beb-6e7c-4afe-add4-a195469fee9a\") " Jul 16 22:36:42.595946 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:42.595928 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf384beb-6e7c-4afe-add4-a195469fee9a-istio-token\") pod \"cf384beb-6e7c-4afe-add4-a195469fee9a\" (UID: \"cf384beb-6e7c-4afe-add4-a195469fee9a\") " Jul 16 22:36:42.596091 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:42.595972 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf384beb-6e7c-4afe-add4-a195469fee9a-workload-certs\") pod \"cf384beb-6e7c-4afe-add4-a195469fee9a\" (UID: \"cf384beb-6e7c-4afe-add4-a195469fee9a\") " Jul 16 22:36:42.596142 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:42.596086 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf384beb-6e7c-4afe-add4-a195469fee9a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cf384beb-6e7c-4afe-add4-a195469fee9a" (UID: "cf384beb-6e7c-4afe-add4-a195469fee9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:36:42.596404 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:42.596348 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf384beb-6e7c-4afe-add4-a195469fee9a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cf384beb-6e7c-4afe-add4-a195469fee9a" (UID: "cf384beb-6e7c-4afe-add4-a195469fee9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:36:42.596404 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:42.596366 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf384beb-6e7c-4afe-add4-a195469fee9a-istio-data" (OuterVolumeSpecName: "istio-data") pod "cf384beb-6e7c-4afe-add4-a195469fee9a" (UID: "cf384beb-6e7c-4afe-add4-a195469fee9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:36:42.596404 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:42.596350 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf384beb-6e7c-4afe-add4-a195469fee9a-workload-socket\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:36:42.596404 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:42.596384 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf384beb-6e7c-4afe-add4-a195469fee9a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cf384beb-6e7c-4afe-add4-a195469fee9a" (UID: "cf384beb-6e7c-4afe-add4-a195469fee9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 22:36:42.596404 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:42.596405 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf384beb-6e7c-4afe-add4-a195469fee9a-credential-socket\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:36:42.597981 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:42.597958 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf384beb-6e7c-4afe-add4-a195469fee9a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cf384beb-6e7c-4afe-add4-a195469fee9a" (UID: "cf384beb-6e7c-4afe-add4-a195469fee9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:36:42.598081 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:42.598064 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf384beb-6e7c-4afe-add4-a195469fee9a-istio-token" (OuterVolumeSpecName: "istio-token") pod "cf384beb-6e7c-4afe-add4-a195469fee9a" (UID: "cf384beb-6e7c-4afe-add4-a195469fee9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 22:36:42.598149 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:42.598132 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf384beb-6e7c-4afe-add4-a195469fee9a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cf384beb-6e7c-4afe-add4-a195469fee9a" (UID: "cf384beb-6e7c-4afe-add4-a195469fee9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 22:36:42.598318 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:42.598291 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf384beb-6e7c-4afe-add4-a195469fee9a-kube-api-access-ddh9x" (OuterVolumeSpecName: "kube-api-access-ddh9x") pod "cf384beb-6e7c-4afe-add4-a195469fee9a" (UID: "cf384beb-6e7c-4afe-add4-a195469fee9a"). InnerVolumeSpecName "kube-api-access-ddh9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 22:36:42.697522 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:42.697482 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf384beb-6e7c-4afe-add4-a195469fee9a-istio-token\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:36:42.697522 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:42.697514 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf384beb-6e7c-4afe-add4-a195469fee9a-workload-certs\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:36:42.697522 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:42.697524 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf384beb-6e7c-4afe-add4-a195469fee9a-istio-podinfo\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:36:42.697522 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:42.697532 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf384beb-6e7c-4afe-add4-a195469fee9a-istio-data\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:36:42.697999 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:42.697543 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddh9x\" (UniqueName: \"kubernetes.io/projected/cf384beb-6e7c-4afe-add4-a195469fee9a-kube-api-access-ddh9x\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:36:42.697999 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:42.697553 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf384beb-6e7c-4afe-add4-a195469fee9a-istio-envoy\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:36:42.697999 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:42.697562 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf384beb-6e7c-4afe-add4-a195469fee9a-istiod-ca-cert\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:36:42.750397 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:42.750361 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:36:42.892506 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:42.892475 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8"] Jul 16 22:36:42.898736 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:42.898707 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:36:42.903168 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:42.903143 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-cca4a2-openshift-default-547ff9b87f-56qb8"] Jul 16 22:36:43.934627 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:43.934594 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf384beb-6e7c-4afe-add4-a195469fee9a" path="/var/lib/kubelet/pods/cf384beb-6e7c-4afe-add4-a195469fee9a/volumes" Jul 16 22:36:44.431835 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:44.431798 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:36:46.167610 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:46.167572 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:36:46.576071 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:46.576041 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:36:46.859674 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:46.859580 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl"] Jul 16 22:36:46.860682 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:46.860655 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf384beb-6e7c-4afe-add4-a195469fee9a" containerName="istio-proxy" Jul 16 22:36:46.860866 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:46.860853 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf384beb-6e7c-4afe-add4-a195469fee9a" containerName="istio-proxy" Jul 16 22:36:46.861142 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:46.861127 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf384beb-6e7c-4afe-add4-a195469fee9a" containerName="istio-proxy" Jul 16 22:36:46.861259 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:46.861248 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf384beb-6e7c-4afe-add4-a195469fee9a" containerName="istio-proxy" Jul 16 22:36:46.867209 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:46.867185 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" Jul 16 22:36:46.871574 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:46.871545 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-1f2f96-openshift-default-dockercfg-zqgpd\"" Jul 16 22:36:46.874731 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:46.874707 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl"] Jul 16 22:36:46.886188 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:46.886166 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:36:47.053580 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:47.053538 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-workload-certs\") pod \"e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl\" (UID: \"49dc4ef4-bde7-4259-b4d7-13ca3950f05e\") " pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" Jul 16 22:36:47.053768 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:47.053591 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-istio-podinfo\") pod \"e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl\" (UID: \"49dc4ef4-bde7-4259-b4d7-13ca3950f05e\") " pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" Jul 16 22:36:47.053768 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:47.053625 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-istio-token\") pod \"e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl\" (UID: \"49dc4ef4-bde7-4259-b4d7-13ca3950f05e\") " pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" Jul 16 22:36:47.053768 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:47.053652 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-workload-socket\") pod \"e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl\" (UID: \"49dc4ef4-bde7-4259-b4d7-13ca3950f05e\") " pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" Jul 16 22:36:47.053768 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:47.053675 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-istio-data\") pod \"e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl\" (UID: \"49dc4ef4-bde7-4259-b4d7-13ca3950f05e\") " pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" Jul 16 22:36:47.053768 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:47.053748 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scfrl\" (UniqueName: \"kubernetes.io/projected/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-kube-api-access-scfrl\") pod \"e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl\" (UID: \"49dc4ef4-bde7-4259-b4d7-13ca3950f05e\") " pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" Jul 16 22:36:47.054029 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:47.053838 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-credential-socket\") pod \"e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl\" (UID: \"49dc4ef4-bde7-4259-b4d7-13ca3950f05e\") " pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" Jul 16 22:36:47.054029 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:47.053864 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-istio-envoy\") pod \"e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl\" (UID: \"49dc4ef4-bde7-4259-b4d7-13ca3950f05e\") " pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" Jul 16 22:36:47.054029 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:47.053948 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-istiod-ca-cert\") pod \"e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl\" (UID: \"49dc4ef4-bde7-4259-b4d7-13ca3950f05e\") " pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" Jul 16 22:36:47.155367 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:47.155273 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-credential-socket\") pod \"e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl\" (UID: \"49dc4ef4-bde7-4259-b4d7-13ca3950f05e\") " pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" Jul 16 22:36:47.155367 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:47.155309 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-istio-envoy\") pod \"e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl\" (UID: \"49dc4ef4-bde7-4259-b4d7-13ca3950f05e\") " pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" Jul 16 22:36:47.155367 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:47.155345 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-istiod-ca-cert\") pod \"e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl\" (UID: \"49dc4ef4-bde7-4259-b4d7-13ca3950f05e\") " pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" Jul 16 22:36:47.155656 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:47.155382 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-workload-certs\") pod \"e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl\" (UID: \"49dc4ef4-bde7-4259-b4d7-13ca3950f05e\") " pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" Jul 16 22:36:47.155656 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:47.155417 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-istio-podinfo\") pod \"e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl\" (UID: \"49dc4ef4-bde7-4259-b4d7-13ca3950f05e\") " pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" Jul 16 22:36:47.155656 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:47.155439 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-istio-token\") pod \"e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl\" (UID: \"49dc4ef4-bde7-4259-b4d7-13ca3950f05e\") " pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" Jul 16 22:36:47.155656 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:47.155467 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-workload-socket\") pod \"e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl\" (UID: \"49dc4ef4-bde7-4259-b4d7-13ca3950f05e\") " pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" Jul 16 22:36:47.155656 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:47.155486 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-istio-data\") pod \"e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl\" (UID: \"49dc4ef4-bde7-4259-b4d7-13ca3950f05e\") " pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" Jul 16 22:36:47.155656 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:47.155524 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scfrl\" (UniqueName: \"kubernetes.io/projected/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-kube-api-access-scfrl\") pod \"e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl\" (UID: \"49dc4ef4-bde7-4259-b4d7-13ca3950f05e\") " pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" Jul 16 22:36:47.155974 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:47.155746 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-credential-socket\") pod \"e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl\" (UID: \"49dc4ef4-bde7-4259-b4d7-13ca3950f05e\") " pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" Jul 16 22:36:47.155974 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:47.155811 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-workload-certs\") pod \"e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl\" (UID: \"49dc4ef4-bde7-4259-b4d7-13ca3950f05e\") " pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" Jul 16 22:36:47.155974 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:47.155935 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-workload-socket\") pod \"e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl\" (UID: \"49dc4ef4-bde7-4259-b4d7-13ca3950f05e\") " pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" Jul 16 22:36:47.156118 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:47.156014 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-istio-data\") pod \"e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl\" (UID: \"49dc4ef4-bde7-4259-b4d7-13ca3950f05e\") " pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" Jul 16 22:36:47.156513 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:47.156490 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-istiod-ca-cert\") pod \"e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl\" (UID: \"49dc4ef4-bde7-4259-b4d7-13ca3950f05e\") " pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" Jul 16 22:36:47.157863 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:47.157834 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-istio-envoy\") pod \"e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl\" (UID: \"49dc4ef4-bde7-4259-b4d7-13ca3950f05e\") " pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" Jul 16 22:36:47.158156 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:47.158133 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-istio-podinfo\") pod \"e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl\" (UID: \"49dc4ef4-bde7-4259-b4d7-13ca3950f05e\") " pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" Jul 16 22:36:47.170306 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:47.170273 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-istio-token\") pod \"e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl\" (UID: \"49dc4ef4-bde7-4259-b4d7-13ca3950f05e\") " pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" Jul 16 22:36:47.170599 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:47.170489 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scfrl\" (UniqueName: \"kubernetes.io/projected/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-kube-api-access-scfrl\") pod \"e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl\" (UID: \"49dc4ef4-bde7-4259-b4d7-13ca3950f05e\") " pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" Jul 16 22:36:47.180363 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:47.180346 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" Jul 16 22:36:47.416111 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:47.416085 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl"] Jul 16 22:36:47.416672 ip-10-0-128-255 kubenswrapper[2585]: W0716 22:36:47.416645 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49dc4ef4_bde7_4259_b4d7_13ca3950f05e.slice/crio-41877a066b9ddbd00508954ade4a2b866e166ad58cc9dffcec3aba5486614b3e WatchSource:0}: Error finding container 41877a066b9ddbd00508954ade4a2b866e166ad58cc9dffcec3aba5486614b3e: Status 404 returned error can't find the container with id 41877a066b9ddbd00508954ade4a2b866e166ad58cc9dffcec3aba5486614b3e Jul 16 22:36:47.421290 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:47.419284 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 22:36:47.421290 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:47.419367 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 22:36:47.421290 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:47.419410 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 22:36:47.560760 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:47.560714 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" event={"ID":"49dc4ef4-bde7-4259-b4d7-13ca3950f05e","Type":"ContainerStarted","Data":"97c8d21652f5a834e737f806f64adb8e16bbf6d12bc7b2183aac6df95767b221"} Jul 16 22:36:47.560917 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:47.560780 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" event={"ID":"49dc4ef4-bde7-4259-b4d7-13ca3950f05e","Type":"ContainerStarted","Data":"41877a066b9ddbd00508954ade4a2b866e166ad58cc9dffcec3aba5486614b3e"} Jul 16 22:36:47.619660 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:47.619594 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" podStartSLOduration=1.619572746 podStartE2EDuration="1.619572746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:36:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:36:47.617924987 +0000 UTC m=+2956.240367941" watchObservedRunningTime="2026-07-16 22:36:47.619572746 +0000 UTC m=+2956.242015701" Jul 16 22:36:47.677746 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:47.677658 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:36:48.180914 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:48.180880 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" Jul 16 22:36:48.185796 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:48.185765 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" Jul 16 22:36:48.566062 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:48.566034 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" Jul 16 22:36:48.567120 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:48.567099 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" Jul 16 22:36:49.678188 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:49.678155 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:36:49.781694 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:49.781662 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:36:49.901258 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:49.901205 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:36:50.013534 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:50.013449 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:36:50.451273 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:50.451238 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:36:50.552810 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:50.552782 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:36:53.596382 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:53.596349 2585 generic.go:356] "Generic (PLEG): container finished" podID="a917555a-1827-4bf1-9958-b76e9ae915ef" containerID="5e5fe933f972cbbdfc2f7dff12b75acb10a6116c1b43f8723648985addfc867c" exitCode=0 Jul 16 22:36:53.596843 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:53.596412 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" event={"ID":"a917555a-1827-4bf1-9958-b76e9ae915ef","Type":"ContainerDied","Data":"5e5fe933f972cbbdfc2f7dff12b75acb10a6116c1b43f8723648985addfc867c"} Jul 16 22:36:53.596893 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:53.596860 2585 scope.go:117] "RemoveContainer" containerID="5e5fe933f972cbbdfc2f7dff12b75acb10a6116c1b43f8723648985addfc867c" Jul 16 22:36:53.597841 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:53.597811 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 22:36:53.597949 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:53.597879 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 22:36:53.597949 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:53.597909 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 22:36:54.079128 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:54.079097 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:36:54.603512 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:54.603478 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" event={"ID":"a917555a-1827-4bf1-9958-b76e9ae915ef","Type":"ContainerStarted","Data":"563bd5381e488d83ac2c1833c74c1d419e1c401cf9f1678f6aeab2a5a9562e11"} Jul 16 22:36:55.520739 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:55.520691 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" Jul 16 22:36:55.596328 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:55.596295 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" Jul 16 22:36:55.597468 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:55.597439 2585 patch_prober.go:28] interesting pod/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 16 22:36:55.597591 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:55.597497 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" podUID="a917555a-1827-4bf1-9958-b76e9ae915ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 16 22:36:56.597636 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:56.597575 2585 patch_prober.go:28] interesting pod/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 16 22:36:56.598155 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:56.597661 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" podUID="a917555a-1827-4bf1-9958-b76e9ae915ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 16 22:36:57.597529 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:57.597488 2585 patch_prober.go:28] interesting pod/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 16 22:36:57.597708 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:57.597543 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" podUID="a917555a-1827-4bf1-9958-b76e9ae915ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 16 22:36:57.693211 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:57.693169 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:36:58.601353 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:58.601325 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" Jul 16 22:36:58.602242 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:36:58.602208 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" Jul 16 22:37:02.643541 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:02.643502 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:37:02.742999 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:02.742968 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:37:04.607390 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:04.607354 2585 patch_prober.go:28] interesting pod/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 16 22:37:04.607834 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:04.607410 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" podUID="49dc4ef4-bde7-4259-b4d7-13ca3950f05e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 16 22:37:05.680646 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:05.680610 2585 generic.go:356] "Generic (PLEG): container finished" podID="49dc4ef4-bde7-4259-b4d7-13ca3950f05e" containerID="97c8d21652f5a834e737f806f64adb8e16bbf6d12bc7b2183aac6df95767b221" exitCode=0 Jul 16 22:37:05.681137 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:05.680683 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" event={"ID":"49dc4ef4-bde7-4259-b4d7-13ca3950f05e","Type":"ContainerDied","Data":"97c8d21652f5a834e737f806f64adb8e16bbf6d12bc7b2183aac6df95767b221"} Jul 16 22:37:05.681137 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:05.681102 2585 scope.go:117] "RemoveContainer" containerID="97c8d21652f5a834e737f806f64adb8e16bbf6d12bc7b2183aac6df95767b221" Jul 16 22:37:05.682120 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:05.682092 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 22:37:05.682197 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:05.682176 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 22:37:05.682259 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:05.682239 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 22:37:06.688322 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:06.688285 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" event={"ID":"49dc4ef4-bde7-4259-b4d7-13ca3950f05e","Type":"ContainerStarted","Data":"cb97f5e32bbb306435172bd2d70119f65c04db68d65de7fa6bc8a7cd88761b4b"} Jul 16 22:37:07.181047 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:07.181009 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" Jul 16 22:37:07.181257 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:07.181213 2585 patch_prober.go:28] interesting pod/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 16 22:37:07.181321 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:07.181294 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" podUID="49dc4ef4-bde7-4259-b4d7-13ca3950f05e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 16 22:37:08.181575 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:08.181535 2585 patch_prober.go:28] interesting pod/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 16 22:37:08.181961 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:08.181595 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" podUID="49dc4ef4-bde7-4259-b4d7-13ca3950f05e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 16 22:37:09.181177 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:09.181137 2585 patch_prober.go:28] interesting pod/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 16 22:37:09.181386 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:09.181198 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" podUID="49dc4ef4-bde7-4259-b4d7-13ca3950f05e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 16 22:37:10.181048 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:10.181011 2585 patch_prober.go:28] interesting pod/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 16 22:37:10.181465 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:10.181077 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" podUID="49dc4ef4-bde7-4259-b4d7-13ca3950f05e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 16 22:37:11.184906 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:11.184873 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" Jul 16 22:37:11.185337 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:11.185193 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" Jul 16 22:37:11.185965 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:11.185948 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" Jul 16 22:37:11.420786 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:11.420755 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:37:11.525372 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:11.525291 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:37:12.198868 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:12.198837 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:37:15.306977 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:15.306942 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:37:17.154049 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:17.154021 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:37:23.342730 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:23.342691 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:37:23.446811 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:23.446776 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:37:27.853644 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:27.853613 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd"] Jul 16 22:37:27.854072 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:27.853843 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" podUID="a917555a-1827-4bf1-9958-b76e9ae915ef" containerName="istio-proxy" containerID="cri-o://563bd5381e488d83ac2c1833c74c1d419e1c401cf9f1678f6aeab2a5a9562e11" gracePeriod=30 Jul 16 22:37:27.878035 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:27.878006 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:37:27.977267 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:27.977208 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:37:28.085069 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:28.085031 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:37:28.604577 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:28.604534 2585 patch_prober.go:28] interesting pod/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 16 22:37:28.604757 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:28.604603 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" podUID="a917555a-1827-4bf1-9958-b76e9ae915ef" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:37:33.095135 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:33.095112 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" Jul 16 22:37:33.194431 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:33.194346 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a917555a-1827-4bf1-9958-b76e9ae915ef-credential-socket\") pod \"a917555a-1827-4bf1-9958-b76e9ae915ef\" (UID: \"a917555a-1827-4bf1-9958-b76e9ae915ef\") " Jul 16 22:37:33.194431 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:33.194391 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a917555a-1827-4bf1-9958-b76e9ae915ef-istio-envoy\") pod \"a917555a-1827-4bf1-9958-b76e9ae915ef\" (UID: \"a917555a-1827-4bf1-9958-b76e9ae915ef\") " Jul 16 22:37:33.194665 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:33.194475 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a917555a-1827-4bf1-9958-b76e9ae915ef-istio-data\") pod \"a917555a-1827-4bf1-9958-b76e9ae915ef\" (UID: \"a917555a-1827-4bf1-9958-b76e9ae915ef\") " Jul 16 22:37:33.194665 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:33.194491 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a917555a-1827-4bf1-9958-b76e9ae915ef-workload-socket\") pod \"a917555a-1827-4bf1-9958-b76e9ae915ef\" (UID: \"a917555a-1827-4bf1-9958-b76e9ae915ef\") " Jul 16 22:37:33.194665 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:33.194510 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pj4l7\" (UniqueName: \"kubernetes.io/projected/a917555a-1827-4bf1-9958-b76e9ae915ef-kube-api-access-pj4l7\") pod \"a917555a-1827-4bf1-9958-b76e9ae915ef\" (UID: \"a917555a-1827-4bf1-9958-b76e9ae915ef\") " Jul 16 22:37:33.194665 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:33.194532 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a917555a-1827-4bf1-9958-b76e9ae915ef-istio-token\") pod \"a917555a-1827-4bf1-9958-b76e9ae915ef\" (UID: \"a917555a-1827-4bf1-9958-b76e9ae915ef\") " Jul 16 22:37:33.194665 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:33.194586 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a917555a-1827-4bf1-9958-b76e9ae915ef-istio-podinfo\") pod \"a917555a-1827-4bf1-9958-b76e9ae915ef\" (UID: \"a917555a-1827-4bf1-9958-b76e9ae915ef\") " Jul 16 22:37:33.194665 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:33.194612 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a917555a-1827-4bf1-9958-b76e9ae915ef-workload-certs\") pod \"a917555a-1827-4bf1-9958-b76e9ae915ef\" (UID: \"a917555a-1827-4bf1-9958-b76e9ae915ef\") " Jul 16 22:37:33.194974 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:33.194659 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a917555a-1827-4bf1-9958-b76e9ae915ef-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a917555a-1827-4bf1-9958-b76e9ae915ef" (UID: "a917555a-1827-4bf1-9958-b76e9ae915ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:37:33.194974 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:33.194717 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a917555a-1827-4bf1-9958-b76e9ae915ef-istiod-ca-cert\") pod \"a917555a-1827-4bf1-9958-b76e9ae915ef\" (UID: \"a917555a-1827-4bf1-9958-b76e9ae915ef\") " Jul 16 22:37:33.194974 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:33.194794 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a917555a-1827-4bf1-9958-b76e9ae915ef-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a917555a-1827-4bf1-9958-b76e9ae915ef" (UID: "a917555a-1827-4bf1-9958-b76e9ae915ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:37:33.195139 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:33.195034 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a917555a-1827-4bf1-9958-b76e9ae915ef-credential-socket\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:37:33.195139 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:33.195050 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a917555a-1827-4bf1-9958-b76e9ae915ef-workload-socket\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:37:33.195139 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:33.195117 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a917555a-1827-4bf1-9958-b76e9ae915ef-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a917555a-1827-4bf1-9958-b76e9ae915ef" (UID: "a917555a-1827-4bf1-9958-b76e9ae915ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:37:33.195302 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:33.195197 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a917555a-1827-4bf1-9958-b76e9ae915ef-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a917555a-1827-4bf1-9958-b76e9ae915ef" (UID: "a917555a-1827-4bf1-9958-b76e9ae915ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 22:37:33.195340 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:33.195298 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a917555a-1827-4bf1-9958-b76e9ae915ef-istio-data" (OuterVolumeSpecName: "istio-data") pod "a917555a-1827-4bf1-9958-b76e9ae915ef" (UID: "a917555a-1827-4bf1-9958-b76e9ae915ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:37:33.196772 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:33.196732 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a917555a-1827-4bf1-9958-b76e9ae915ef-kube-api-access-pj4l7" (OuterVolumeSpecName: "kube-api-access-pj4l7") pod "a917555a-1827-4bf1-9958-b76e9ae915ef" (UID: "a917555a-1827-4bf1-9958-b76e9ae915ef"). InnerVolumeSpecName "kube-api-access-pj4l7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 22:37:33.196866 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:33.196843 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a917555a-1827-4bf1-9958-b76e9ae915ef-istio-token" (OuterVolumeSpecName: "istio-token") pod "a917555a-1827-4bf1-9958-b76e9ae915ef" (UID: "a917555a-1827-4bf1-9958-b76e9ae915ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 22:37:33.197055 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:33.197039 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a917555a-1827-4bf1-9958-b76e9ae915ef-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a917555a-1827-4bf1-9958-b76e9ae915ef" (UID: "a917555a-1827-4bf1-9958-b76e9ae915ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 22:37:33.197145 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:33.197130 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a917555a-1827-4bf1-9958-b76e9ae915ef-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a917555a-1827-4bf1-9958-b76e9ae915ef" (UID: "a917555a-1827-4bf1-9958-b76e9ae915ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:37:33.295536 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:33.295491 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a917555a-1827-4bf1-9958-b76e9ae915ef-istio-podinfo\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:37:33.295536 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:33.295532 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a917555a-1827-4bf1-9958-b76e9ae915ef-workload-certs\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:37:33.295536 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:33.295545 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a917555a-1827-4bf1-9958-b76e9ae915ef-istiod-ca-cert\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:37:33.295766 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:33.295557 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a917555a-1827-4bf1-9958-b76e9ae915ef-istio-envoy\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:37:33.295766 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:33.295569 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a917555a-1827-4bf1-9958-b76e9ae915ef-istio-data\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:37:33.295766 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:33.295581 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj4l7\" (UniqueName: \"kubernetes.io/projected/a917555a-1827-4bf1-9958-b76e9ae915ef-kube-api-access-pj4l7\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:37:33.295766 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:33.295593 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a917555a-1827-4bf1-9958-b76e9ae915ef-istio-token\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:37:33.830957 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:33.830919 2585 generic.go:356] "Generic (PLEG): container finished" podID="a917555a-1827-4bf1-9958-b76e9ae915ef" containerID="563bd5381e488d83ac2c1833c74c1d419e1c401cf9f1678f6aeab2a5a9562e11" exitCode=0 Jul 16 22:37:33.831143 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:33.830971 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" Jul 16 22:37:33.831143 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:33.831005 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" event={"ID":"a917555a-1827-4bf1-9958-b76e9ae915ef","Type":"ContainerDied","Data":"563bd5381e488d83ac2c1833c74c1d419e1c401cf9f1678f6aeab2a5a9562e11"} Jul 16 22:37:33.831143 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:33.831059 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd" event={"ID":"a917555a-1827-4bf1-9958-b76e9ae915ef","Type":"ContainerDied","Data":"94be9b291149507bde91e064e054589d9d686b72431b3e4b6c1a5d769affb7ac"} Jul 16 22:37:33.831143 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:33.831081 2585 scope.go:117] "RemoveContainer" containerID="563bd5381e488d83ac2c1833c74c1d419e1c401cf9f1678f6aeab2a5a9562e11" Jul 16 22:37:33.840400 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:33.840383 2585 scope.go:117] "RemoveContainer" containerID="5e5fe933f972cbbdfc2f7dff12b75acb10a6116c1b43f8723648985addfc867c" Jul 16 22:37:33.848171 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:33.848151 2585 scope.go:117] "RemoveContainer" containerID="563bd5381e488d83ac2c1833c74c1d419e1c401cf9f1678f6aeab2a5a9562e11" Jul 16 22:37:33.848463 ip-10-0-128-255 kubenswrapper[2585]: E0716 22:37:33.848437 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"563bd5381e488d83ac2c1833c74c1d419e1c401cf9f1678f6aeab2a5a9562e11\": container with ID starting with 563bd5381e488d83ac2c1833c74c1d419e1c401cf9f1678f6aeab2a5a9562e11 not found: ID does not exist" containerID="563bd5381e488d83ac2c1833c74c1d419e1c401cf9f1678f6aeab2a5a9562e11" Jul 16 22:37:33.848523 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:33.848476 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"563bd5381e488d83ac2c1833c74c1d419e1c401cf9f1678f6aeab2a5a9562e11"} err="failed to get container status \"563bd5381e488d83ac2c1833c74c1d419e1c401cf9f1678f6aeab2a5a9562e11\": rpc error: code = NotFound desc = could not find container \"563bd5381e488d83ac2c1833c74c1d419e1c401cf9f1678f6aeab2a5a9562e11\": container with ID starting with 563bd5381e488d83ac2c1833c74c1d419e1c401cf9f1678f6aeab2a5a9562e11 not found: ID does not exist" Jul 16 22:37:33.848523 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:33.848494 2585 scope.go:117] "RemoveContainer" containerID="5e5fe933f972cbbdfc2f7dff12b75acb10a6116c1b43f8723648985addfc867c" Jul 16 22:37:33.848710 ip-10-0-128-255 kubenswrapper[2585]: E0716 22:37:33.848688 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e5fe933f972cbbdfc2f7dff12b75acb10a6116c1b43f8723648985addfc867c\": container with ID starting with 5e5fe933f972cbbdfc2f7dff12b75acb10a6116c1b43f8723648985addfc867c not found: ID does not exist" containerID="5e5fe933f972cbbdfc2f7dff12b75acb10a6116c1b43f8723648985addfc867c" Jul 16 22:37:33.848753 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:33.848714 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e5fe933f972cbbdfc2f7dff12b75acb10a6116c1b43f8723648985addfc867c"} err="failed to get container status \"5e5fe933f972cbbdfc2f7dff12b75acb10a6116c1b43f8723648985addfc867c\": rpc error: code = NotFound desc = could not find container \"5e5fe933f972cbbdfc2f7dff12b75acb10a6116c1b43f8723648985addfc867c\": container with ID starting with 5e5fe933f972cbbdfc2f7dff12b75acb10a6116c1b43f8723648985addfc867c not found: ID does not exist" Jul 16 22:37:33.853793 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:33.853773 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd"] Jul 16 22:37:33.857705 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:33.857684 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-8247b5-openshift-default-66cddc84c5-gvkvd"] Jul 16 22:37:33.935118 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:33.935093 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a917555a-1827-4bf1-9958-b76e9ae915ef" path="/var/lib/kubelet/pods/a917555a-1827-4bf1-9958-b76e9ae915ef/volumes" Jul 16 22:37:38.803764 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:38.803728 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:37:42.197767 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:42.197726 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:37:43.328964 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:43.328928 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:37:48.213054 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:48.213020 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:37:48.409169 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:48.409132 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:37:49.837329 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:49.837278 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:37:49.939283 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:49.939254 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:37:53.980765 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:53.980730 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl"] Jul 16 22:37:53.981180 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:53.980983 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" podUID="49dc4ef4-bde7-4259-b4d7-13ca3950f05e" containerName="istio-proxy" containerID="cri-o://cb97f5e32bbb306435172bd2d70119f65c04db68d65de7fa6bc8a7cd88761b4b" gracePeriod=30 Jul 16 22:37:54.006639 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:54.006613 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:37:54.886732 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:54.886704 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk"] Jul 16 22:37:54.887055 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:54.887043 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a917555a-1827-4bf1-9958-b76e9ae915ef" containerName="istio-proxy" Jul 16 22:37:54.887111 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:54.887056 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a917555a-1827-4bf1-9958-b76e9ae915ef" containerName="istio-proxy" Jul 16 22:37:54.887111 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:54.887073 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf384beb-6e7c-4afe-add4-a195469fee9a" containerName="istio-proxy" Jul 16 22:37:54.887111 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:54.887079 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf384beb-6e7c-4afe-add4-a195469fee9a" containerName="istio-proxy" Jul 16 22:37:54.887230 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:54.887142 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="a917555a-1827-4bf1-9958-b76e9ae915ef" containerName="istio-proxy" Jul 16 22:37:54.887230 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:54.887153 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="a917555a-1827-4bf1-9958-b76e9ae915ef" containerName="istio-proxy" Jul 16 22:37:54.896054 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:54.896027 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" Jul 16 22:37:54.899086 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:54.899061 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-1c9596-openshift-default-dockercfg-nrfjj\"" Jul 16 22:37:54.904012 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:54.903937 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk"] Jul 16 22:37:54.982604 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:54.982571 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b28f8dc-5933-45b9-ad31-94a3492ee25e-istio-data\") pod \"e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk\" (UID: \"8b28f8dc-5933-45b9-ad31-94a3492ee25e\") " pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" Jul 16 22:37:54.982604 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:54.982607 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b28f8dc-5933-45b9-ad31-94a3492ee25e-workload-socket\") pod \"e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk\" (UID: \"8b28f8dc-5933-45b9-ad31-94a3492ee25e\") " pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" Jul 16 22:37:54.983127 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:54.982734 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b28f8dc-5933-45b9-ad31-94a3492ee25e-istio-token\") pod \"e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk\" (UID: \"8b28f8dc-5933-45b9-ad31-94a3492ee25e\") " pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" Jul 16 22:37:54.983127 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:54.982782 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b28f8dc-5933-45b9-ad31-94a3492ee25e-istio-podinfo\") pod \"e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk\" (UID: \"8b28f8dc-5933-45b9-ad31-94a3492ee25e\") " pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" Jul 16 22:37:54.983127 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:54.982807 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b28f8dc-5933-45b9-ad31-94a3492ee25e-istiod-ca-cert\") pod \"e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk\" (UID: \"8b28f8dc-5933-45b9-ad31-94a3492ee25e\") " pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" Jul 16 22:37:54.983127 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:54.982853 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx8jb\" (UniqueName: \"kubernetes.io/projected/8b28f8dc-5933-45b9-ad31-94a3492ee25e-kube-api-access-jx8jb\") pod \"e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk\" (UID: \"8b28f8dc-5933-45b9-ad31-94a3492ee25e\") " pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" Jul 16 22:37:54.983127 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:54.982894 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b28f8dc-5933-45b9-ad31-94a3492ee25e-workload-certs\") pod \"e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk\" (UID: \"8b28f8dc-5933-45b9-ad31-94a3492ee25e\") " pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" Jul 16 22:37:54.983127 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:54.982948 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b28f8dc-5933-45b9-ad31-94a3492ee25e-credential-socket\") pod \"e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk\" (UID: \"8b28f8dc-5933-45b9-ad31-94a3492ee25e\") " pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" Jul 16 22:37:54.983127 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:54.982988 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b28f8dc-5933-45b9-ad31-94a3492ee25e-istio-envoy\") pod \"e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk\" (UID: \"8b28f8dc-5933-45b9-ad31-94a3492ee25e\") " pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" Jul 16 22:37:55.083872 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:55.083833 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b28f8dc-5933-45b9-ad31-94a3492ee25e-istio-token\") pod \"e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk\" (UID: \"8b28f8dc-5933-45b9-ad31-94a3492ee25e\") " pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" Jul 16 22:37:55.084050 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:55.083880 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b28f8dc-5933-45b9-ad31-94a3492ee25e-istio-podinfo\") pod \"e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk\" (UID: \"8b28f8dc-5933-45b9-ad31-94a3492ee25e\") " pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" Jul 16 22:37:55.084050 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:55.083902 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b28f8dc-5933-45b9-ad31-94a3492ee25e-istiod-ca-cert\") pod \"e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk\" (UID: \"8b28f8dc-5933-45b9-ad31-94a3492ee25e\") " pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" Jul 16 22:37:55.084050 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:55.083939 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx8jb\" (UniqueName: \"kubernetes.io/projected/8b28f8dc-5933-45b9-ad31-94a3492ee25e-kube-api-access-jx8jb\") pod \"e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk\" (UID: \"8b28f8dc-5933-45b9-ad31-94a3492ee25e\") " pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" Jul 16 22:37:55.084050 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:55.083962 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b28f8dc-5933-45b9-ad31-94a3492ee25e-workload-certs\") pod \"e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk\" (UID: \"8b28f8dc-5933-45b9-ad31-94a3492ee25e\") " pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" Jul 16 22:37:55.084050 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:55.083995 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b28f8dc-5933-45b9-ad31-94a3492ee25e-credential-socket\") pod \"e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk\" (UID: \"8b28f8dc-5933-45b9-ad31-94a3492ee25e\") " pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" Jul 16 22:37:55.084050 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:55.084031 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b28f8dc-5933-45b9-ad31-94a3492ee25e-istio-envoy\") pod \"e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk\" (UID: \"8b28f8dc-5933-45b9-ad31-94a3492ee25e\") " pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" Jul 16 22:37:55.084296 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:55.084073 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b28f8dc-5933-45b9-ad31-94a3492ee25e-istio-data\") pod \"e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk\" (UID: \"8b28f8dc-5933-45b9-ad31-94a3492ee25e\") " pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" Jul 16 22:37:55.084296 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:55.084101 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b28f8dc-5933-45b9-ad31-94a3492ee25e-workload-socket\") pod \"e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk\" (UID: \"8b28f8dc-5933-45b9-ad31-94a3492ee25e\") " pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" Jul 16 22:37:55.084560 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:55.084508 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b28f8dc-5933-45b9-ad31-94a3492ee25e-workload-socket\") pod \"e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk\" (UID: \"8b28f8dc-5933-45b9-ad31-94a3492ee25e\") " pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" Jul 16 22:37:55.084703 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:55.084660 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b28f8dc-5933-45b9-ad31-94a3492ee25e-credential-socket\") pod \"e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk\" (UID: \"8b28f8dc-5933-45b9-ad31-94a3492ee25e\") " pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" Jul 16 22:37:55.084774 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:55.084710 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b28f8dc-5933-45b9-ad31-94a3492ee25e-workload-certs\") pod \"e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk\" (UID: \"8b28f8dc-5933-45b9-ad31-94a3492ee25e\") " pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" Jul 16 22:37:55.084820 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:55.084783 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b28f8dc-5933-45b9-ad31-94a3492ee25e-istiod-ca-cert\") pod \"e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk\" (UID: \"8b28f8dc-5933-45b9-ad31-94a3492ee25e\") " pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" Jul 16 22:37:55.084820 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:55.084798 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b28f8dc-5933-45b9-ad31-94a3492ee25e-istio-data\") pod \"e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk\" (UID: \"8b28f8dc-5933-45b9-ad31-94a3492ee25e\") " pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" Jul 16 22:37:55.086556 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:55.086535 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b28f8dc-5933-45b9-ad31-94a3492ee25e-istio-podinfo\") pod \"e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk\" (UID: \"8b28f8dc-5933-45b9-ad31-94a3492ee25e\") " pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" Jul 16 22:37:55.086726 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:55.086705 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b28f8dc-5933-45b9-ad31-94a3492ee25e-istio-envoy\") pod \"e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk\" (UID: \"8b28f8dc-5933-45b9-ad31-94a3492ee25e\") " pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" Jul 16 22:37:55.093070 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:55.093047 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx8jb\" (UniqueName: \"kubernetes.io/projected/8b28f8dc-5933-45b9-ad31-94a3492ee25e-kube-api-access-jx8jb\") pod \"e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk\" (UID: \"8b28f8dc-5933-45b9-ad31-94a3492ee25e\") " pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" Jul 16 22:37:55.093167 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:55.093119 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b28f8dc-5933-45b9-ad31-94a3492ee25e-istio-token\") pod \"e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk\" (UID: \"8b28f8dc-5933-45b9-ad31-94a3492ee25e\") " pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" Jul 16 22:37:55.208443 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:55.208346 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" Jul 16 22:37:55.358831 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:55.358807 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk"] Jul 16 22:37:55.360943 ip-10-0-128-255 kubenswrapper[2585]: W0716 22:37:55.360912 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b28f8dc_5933_45b9_ad31_94a3492ee25e.slice/crio-b67f04e998ef6176a001d85c773d553fc0da48530e044fc43c97ae85ac12e239 WatchSource:0}: Error finding container b67f04e998ef6176a001d85c773d553fc0da48530e044fc43c97ae85ac12e239: Status 404 returned error can't find the container with id b67f04e998ef6176a001d85c773d553fc0da48530e044fc43c97ae85ac12e239 Jul 16 22:37:55.363158 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:55.363123 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 22:37:55.363254 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:55.363203 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 22:37:55.363305 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:55.363271 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 22:37:55.474350 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:55.474263 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:37:55.599881 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:55.599828 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:37:55.945587 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:55.945551 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" event={"ID":"8b28f8dc-5933-45b9-ad31-94a3492ee25e","Type":"ContainerStarted","Data":"3ca92af51e7cc51369554cd6f6f7fb0e4d810d25e201b3ad03377b84b4804597"} Jul 16 22:37:55.945736 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:55.945604 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" event={"ID":"8b28f8dc-5933-45b9-ad31-94a3492ee25e","Type":"ContainerStarted","Data":"b67f04e998ef6176a001d85c773d553fc0da48530e044fc43c97ae85ac12e239"} Jul 16 22:37:55.970161 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:55.970113 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" podStartSLOduration=1.970098751 podStartE2EDuration="1.970098751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:37:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:37:55.968240803 +0000 UTC m=+3024.590683751" watchObservedRunningTime="2026-07-16 22:37:55.970098751 +0000 UTC m=+3024.592541703" Jul 16 22:37:56.187640 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:56.187603 2585 patch_prober.go:28] interesting pod/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 16 22:37:56.187982 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:56.187663 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" podUID="49dc4ef4-bde7-4259-b4d7-13ca3950f05e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:37:56.209106 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:56.209038 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" Jul 16 22:37:56.213335 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:56.213312 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" Jul 16 22:37:56.951241 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:56.951190 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" Jul 16 22:37:56.952159 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:56.952139 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" Jul 16 22:37:57.941621 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:57.941585 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:37:58.061968 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:58.061941 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:37:58.165317 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:58.165271 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:37:58.362955 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:58.362927 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:37:58.692929 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:58.692838 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:37:58.792628 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:58.792599 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:37:59.229474 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:59.229453 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" Jul 16 22:37:59.319634 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:59.319608 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-scfrl\" (UniqueName: \"kubernetes.io/projected/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-kube-api-access-scfrl\") pod \"49dc4ef4-bde7-4259-b4d7-13ca3950f05e\" (UID: \"49dc4ef4-bde7-4259-b4d7-13ca3950f05e\") " Jul 16 22:37:59.319785 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:59.319641 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-istio-envoy\") pod \"49dc4ef4-bde7-4259-b4d7-13ca3950f05e\" (UID: \"49dc4ef4-bde7-4259-b4d7-13ca3950f05e\") " Jul 16 22:37:59.319785 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:59.319669 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-istio-data\") pod \"49dc4ef4-bde7-4259-b4d7-13ca3950f05e\" (UID: \"49dc4ef4-bde7-4259-b4d7-13ca3950f05e\") " Jul 16 22:37:59.319785 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:59.319704 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-istio-podinfo\") pod \"49dc4ef4-bde7-4259-b4d7-13ca3950f05e\" (UID: \"49dc4ef4-bde7-4259-b4d7-13ca3950f05e\") " Jul 16 22:37:59.319785 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:59.319728 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-istiod-ca-cert\") pod \"49dc4ef4-bde7-4259-b4d7-13ca3950f05e\" (UID: \"49dc4ef4-bde7-4259-b4d7-13ca3950f05e\") " Jul 16 22:37:59.319785 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:59.319757 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-workload-certs\") pod \"49dc4ef4-bde7-4259-b4d7-13ca3950f05e\" (UID: \"49dc4ef4-bde7-4259-b4d7-13ca3950f05e\") " Jul 16 22:37:59.320046 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:59.319801 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-credential-socket\") pod \"49dc4ef4-bde7-4259-b4d7-13ca3950f05e\" (UID: \"49dc4ef4-bde7-4259-b4d7-13ca3950f05e\") " Jul 16 22:37:59.320046 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:59.319831 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-workload-socket\") pod \"49dc4ef4-bde7-4259-b4d7-13ca3950f05e\" (UID: \"49dc4ef4-bde7-4259-b4d7-13ca3950f05e\") " Jul 16 22:37:59.320046 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:59.319914 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-istio-token\") pod \"49dc4ef4-bde7-4259-b4d7-13ca3950f05e\" (UID: \"49dc4ef4-bde7-4259-b4d7-13ca3950f05e\") " Jul 16 22:37:59.320205 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:59.320106 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "49dc4ef4-bde7-4259-b4d7-13ca3950f05e" (UID: "49dc4ef4-bde7-4259-b4d7-13ca3950f05e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:37:59.320205 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:59.320182 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "49dc4ef4-bde7-4259-b4d7-13ca3950f05e" (UID: "49dc4ef4-bde7-4259-b4d7-13ca3950f05e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 22:37:59.320355 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:59.320342 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-credential-socket\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:37:59.320455 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:59.320427 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-istiod-ca-cert\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:37:59.320455 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:59.320440 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "49dc4ef4-bde7-4259-b4d7-13ca3950f05e" (UID: "49dc4ef4-bde7-4259-b4d7-13ca3950f05e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:37:59.320668 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:59.320643 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "49dc4ef4-bde7-4259-b4d7-13ca3950f05e" (UID: "49dc4ef4-bde7-4259-b4d7-13ca3950f05e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:37:59.320836 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:59.320808 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-istio-data" (OuterVolumeSpecName: "istio-data") pod "49dc4ef4-bde7-4259-b4d7-13ca3950f05e" (UID: "49dc4ef4-bde7-4259-b4d7-13ca3950f05e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:37:59.321941 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:59.321917 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "49dc4ef4-bde7-4259-b4d7-13ca3950f05e" (UID: "49dc4ef4-bde7-4259-b4d7-13ca3950f05e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:37:59.322031 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:59.321943 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-kube-api-access-scfrl" (OuterVolumeSpecName: "kube-api-access-scfrl") pod "49dc4ef4-bde7-4259-b4d7-13ca3950f05e" (UID: "49dc4ef4-bde7-4259-b4d7-13ca3950f05e"). InnerVolumeSpecName "kube-api-access-scfrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 22:37:59.322376 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:59.322357 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-istio-token" (OuterVolumeSpecName: "istio-token") pod "49dc4ef4-bde7-4259-b4d7-13ca3950f05e" (UID: "49dc4ef4-bde7-4259-b4d7-13ca3950f05e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 22:37:59.322376 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:59.322358 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "49dc4ef4-bde7-4259-b4d7-13ca3950f05e" (UID: "49dc4ef4-bde7-4259-b4d7-13ca3950f05e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 22:37:59.421030 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:59.420994 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scfrl\" (UniqueName: \"kubernetes.io/projected/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-kube-api-access-scfrl\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:37:59.421030 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:59.421024 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-istio-envoy\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:37:59.421030 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:59.421037 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-istio-data\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:37:59.421281 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:59.421048 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-istio-podinfo\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:37:59.421281 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:59.421060 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-workload-certs\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:37:59.421281 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:59.421071 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-workload-socket\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:37:59.421281 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:59.421081 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49dc4ef4-bde7-4259-b4d7-13ca3950f05e-istio-token\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:37:59.967892 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:59.967861 2585 generic.go:356] "Generic (PLEG): container finished" podID="49dc4ef4-bde7-4259-b4d7-13ca3950f05e" containerID="cb97f5e32bbb306435172bd2d70119f65c04db68d65de7fa6bc8a7cd88761b4b" exitCode=0 Jul 16 22:37:59.968038 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:59.967904 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" Jul 16 22:37:59.968038 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:59.967940 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" event={"ID":"49dc4ef4-bde7-4259-b4d7-13ca3950f05e","Type":"ContainerDied","Data":"cb97f5e32bbb306435172bd2d70119f65c04db68d65de7fa6bc8a7cd88761b4b"} Jul 16 22:37:59.968038 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:59.967992 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl" event={"ID":"49dc4ef4-bde7-4259-b4d7-13ca3950f05e","Type":"ContainerDied","Data":"41877a066b9ddbd00508954ade4a2b866e166ad58cc9dffcec3aba5486614b3e"} Jul 16 22:37:59.968038 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:59.968015 2585 scope.go:117] "RemoveContainer" containerID="cb97f5e32bbb306435172bd2d70119f65c04db68d65de7fa6bc8a7cd88761b4b" Jul 16 22:37:59.977611 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:59.977593 2585 scope.go:117] "RemoveContainer" containerID="97c8d21652f5a834e737f806f64adb8e16bbf6d12bc7b2183aac6df95767b221" Jul 16 22:37:59.986669 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:59.986646 2585 scope.go:117] "RemoveContainer" containerID="cb97f5e32bbb306435172bd2d70119f65c04db68d65de7fa6bc8a7cd88761b4b" Jul 16 22:37:59.986953 ip-10-0-128-255 kubenswrapper[2585]: E0716 22:37:59.986933 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb97f5e32bbb306435172bd2d70119f65c04db68d65de7fa6bc8a7cd88761b4b\": container with ID starting with cb97f5e32bbb306435172bd2d70119f65c04db68d65de7fa6bc8a7cd88761b4b not found: ID does not exist" containerID="cb97f5e32bbb306435172bd2d70119f65c04db68d65de7fa6bc8a7cd88761b4b" Jul 16 22:37:59.987052 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:59.986963 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb97f5e32bbb306435172bd2d70119f65c04db68d65de7fa6bc8a7cd88761b4b"} err="failed to get container status \"cb97f5e32bbb306435172bd2d70119f65c04db68d65de7fa6bc8a7cd88761b4b\": rpc error: code = NotFound desc = could not find container \"cb97f5e32bbb306435172bd2d70119f65c04db68d65de7fa6bc8a7cd88761b4b\": container with ID starting with cb97f5e32bbb306435172bd2d70119f65c04db68d65de7fa6bc8a7cd88761b4b not found: ID does not exist" Jul 16 22:37:59.987052 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:59.986982 2585 scope.go:117] "RemoveContainer" containerID="97c8d21652f5a834e737f806f64adb8e16bbf6d12bc7b2183aac6df95767b221" Jul 16 22:37:59.987202 ip-10-0-128-255 kubenswrapper[2585]: E0716 22:37:59.987186 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97c8d21652f5a834e737f806f64adb8e16bbf6d12bc7b2183aac6df95767b221\": container with ID starting with 97c8d21652f5a834e737f806f64adb8e16bbf6d12bc7b2183aac6df95767b221 not found: ID does not exist" containerID="97c8d21652f5a834e737f806f64adb8e16bbf6d12bc7b2183aac6df95767b221" Jul 16 22:37:59.987264 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:59.987208 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97c8d21652f5a834e737f806f64adb8e16bbf6d12bc7b2183aac6df95767b221"} err="failed to get container status \"97c8d21652f5a834e737f806f64adb8e16bbf6d12bc7b2183aac6df95767b221\": rpc error: code = NotFound desc = could not find container \"97c8d21652f5a834e737f806f64adb8e16bbf6d12bc7b2183aac6df95767b221\": container with ID starting with 97c8d21652f5a834e737f806f64adb8e16bbf6d12bc7b2183aac6df95767b221 not found: ID does not exist" Jul 16 22:37:59.988458 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:37:59.988439 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl"] Jul 16 22:38:00.010310 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:00.010286 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-1f2f96-openshift-default-fd4f4c9c7-9xdtl"] Jul 16 22:38:01.941066 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:01.941030 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49dc4ef4-bde7-4259-b4d7-13ca3950f05e" path="/var/lib/kubelet/pods/49dc4ef4-bde7-4259-b4d7-13ca3950f05e/volumes" Jul 16 22:38:02.414993 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:02.414952 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:38:02.745913 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:02.745816 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:38:03.038643 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.038564 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g"] Jul 16 22:38:03.039113 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.039097 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49dc4ef4-bde7-4259-b4d7-13ca3950f05e" containerName="istio-proxy" Jul 16 22:38:03.039160 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.039116 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="49dc4ef4-bde7-4259-b4d7-13ca3950f05e" containerName="istio-proxy" Jul 16 22:38:03.039160 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.039133 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a917555a-1827-4bf1-9958-b76e9ae915ef" containerName="istio-proxy" Jul 16 22:38:03.039160 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.039142 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a917555a-1827-4bf1-9958-b76e9ae915ef" containerName="istio-proxy" Jul 16 22:38:03.039279 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.039255 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="49dc4ef4-bde7-4259-b4d7-13ca3950f05e" containerName="istio-proxy" Jul 16 22:38:03.039279 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.039269 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="49dc4ef4-bde7-4259-b4d7-13ca3950f05e" containerName="istio-proxy" Jul 16 22:38:03.044072 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.044053 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" Jul 16 22:38:03.055899 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.055879 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-7d2f73-openshift-default-dockercfg-9ks92\"" Jul 16 22:38:03.070162 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.070136 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g"] Jul 16 22:38:03.100013 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.099983 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:38:03.151439 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.151404 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62dxf\" (UniqueName: \"kubernetes.io/projected/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-kube-api-access-62dxf\") pod \"e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g\" (UID: \"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f\") " pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" Jul 16 22:38:03.151657 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.151464 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-credential-socket\") pod \"e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g\" (UID: \"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f\") " pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" Jul 16 22:38:03.151657 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.151562 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-workload-socket\") pod \"e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g\" (UID: \"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f\") " pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" Jul 16 22:38:03.151657 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.151590 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-istiod-ca-cert\") pod \"e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g\" (UID: \"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f\") " pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" Jul 16 22:38:03.151657 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.151630 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-workload-certs\") pod \"e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g\" (UID: \"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f\") " pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" Jul 16 22:38:03.151866 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.151660 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-istio-podinfo\") pod \"e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g\" (UID: \"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f\") " pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" Jul 16 22:38:03.151866 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.151693 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-istio-data\") pod \"e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g\" (UID: \"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f\") " pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" Jul 16 22:38:03.151866 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.151734 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-istio-envoy\") pod \"e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g\" (UID: \"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f\") " pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" Jul 16 22:38:03.151866 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.151764 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-istio-token\") pod \"e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g\" (UID: \"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f\") " pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" Jul 16 22:38:03.211279 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.211247 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:38:03.252166 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.252134 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-istiod-ca-cert\") pod \"e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g\" (UID: \"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f\") " pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" Jul 16 22:38:03.252333 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.252180 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-workload-certs\") pod \"e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g\" (UID: \"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f\") " pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" Jul 16 22:38:03.252333 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.252207 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-istio-podinfo\") pod \"e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g\" (UID: \"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f\") " pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" Jul 16 22:38:03.252414 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.252387 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-istio-data\") pod \"e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g\" (UID: \"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f\") " pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" Jul 16 22:38:03.252449 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.252432 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-istio-envoy\") pod \"e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g\" (UID: \"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f\") " pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" Jul 16 22:38:03.252501 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.252461 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-istio-token\") pod \"e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g\" (UID: \"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f\") " pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" Jul 16 22:38:03.252632 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.252601 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-workload-certs\") pod \"e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g\" (UID: \"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f\") " pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" Jul 16 22:38:03.252813 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.252665 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62dxf\" (UniqueName: \"kubernetes.io/projected/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-kube-api-access-62dxf\") pod \"e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g\" (UID: \"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f\") " pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" Jul 16 22:38:03.252995 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.252763 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-istio-data\") pod \"e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g\" (UID: \"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f\") " pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" Jul 16 22:38:03.253108 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.252969 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-credential-socket\") pod \"e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g\" (UID: \"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f\") " pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" Jul 16 22:38:03.253108 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.253070 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-istiod-ca-cert\") pod \"e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g\" (UID: \"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f\") " pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" Jul 16 22:38:03.253108 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.253090 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-workload-socket\") pod \"e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g\" (UID: \"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f\") " pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" Jul 16 22:38:03.253349 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.253328 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-credential-socket\") pod \"e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g\" (UID: \"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f\") " pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" Jul 16 22:38:03.253507 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.253424 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-workload-socket\") pod \"e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g\" (UID: \"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f\") " pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" Jul 16 22:38:03.254708 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.254686 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-istio-envoy\") pod \"e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g\" (UID: \"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f\") " pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" Jul 16 22:38:03.254826 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.254808 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-istio-podinfo\") pod \"e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g\" (UID: \"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f\") " pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" Jul 16 22:38:03.260859 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.260830 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62dxf\" (UniqueName: \"kubernetes.io/projected/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-kube-api-access-62dxf\") pod \"e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g\" (UID: \"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f\") " pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" Jul 16 22:38:03.261037 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.261021 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-istio-token\") pod \"e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g\" (UID: \"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f\") " pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" Jul 16 22:38:03.354082 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.354049 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" Jul 16 22:38:03.405488 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.405092 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:38:03.492784 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.492748 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g"] Jul 16 22:38:03.495203 ip-10-0-128-255 kubenswrapper[2585]: W0716 22:38:03.495179 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbba4d0a2_ac2b_47f5_b7b5_d7d9a8203c6f.slice/crio-94a2729a767ffbb0594135df7a33e83d18af5827d69757eb62977e260b69f073 WatchSource:0}: Error finding container 94a2729a767ffbb0594135df7a33e83d18af5827d69757eb62977e260b69f073: Status 404 returned error can't find the container with id 94a2729a767ffbb0594135df7a33e83d18af5827d69757eb62977e260b69f073 Jul 16 22:38:03.497410 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.497372 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 22:38:03.497481 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.497453 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 22:38:03.497526 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.497497 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 22:38:03.993816 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.993781 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" event={"ID":"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f","Type":"ContainerStarted","Data":"795b1754ac421823b8c878b7be7d63e8b7db7125a9af3f173c9f7c76bc21bed9"} Jul 16 22:38:03.993992 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:03.993834 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" event={"ID":"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f","Type":"ContainerStarted","Data":"94a2729a767ffbb0594135df7a33e83d18af5827d69757eb62977e260b69f073"} Jul 16 22:38:04.020673 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:04.020627 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" podStartSLOduration=1.020613739 podStartE2EDuration="1.020613739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:38:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:38:04.019661404 +0000 UTC m=+3032.642104357" watchObservedRunningTime="2026-07-16 22:38:04.020613739 +0000 UTC m=+3032.643056692" Jul 16 22:38:04.355174 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:04.355147 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" Jul 16 22:38:04.360419 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:04.360395 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" Jul 16 22:38:04.999338 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:04.999308 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" Jul 16 22:38:05.000296 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:05.000277 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" Jul 16 22:38:06.039505 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:06.039468 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:38:06.182383 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:06.182346 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:38:06.277075 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:06.277041 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:38:06.425318 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:06.425269 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:38:06.970710 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:06.970680 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:38:07.077312 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:07.077280 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:38:10.506152 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:10.506124 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:38:11.952351 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:11.952272 2585 patch_prober.go:28] interesting pod/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 16 22:38:11.952351 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:11.952326 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" podUID="8b28f8dc-5933-45b9-ad31-94a3492ee25e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 16 22:38:12.041901 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:12.041862 2585 generic.go:356] "Generic (PLEG): container finished" podID="8b28f8dc-5933-45b9-ad31-94a3492ee25e" containerID="3ca92af51e7cc51369554cd6f6f7fb0e4d810d25e201b3ad03377b84b4804597" exitCode=0 Jul 16 22:38:12.041901 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:12.041884 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" event={"ID":"8b28f8dc-5933-45b9-ad31-94a3492ee25e","Type":"ContainerDied","Data":"3ca92af51e7cc51369554cd6f6f7fb0e4d810d25e201b3ad03377b84b4804597"} Jul 16 22:38:12.042375 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:12.042353 2585 scope.go:117] "RemoveContainer" containerID="3ca92af51e7cc51369554cd6f6f7fb0e4d810d25e201b3ad03377b84b4804597" Jul 16 22:38:12.043712 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:12.043670 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 22:38:12.043820 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:12.043776 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 22:38:12.043889 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:12.043852 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 22:38:12.751001 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:12.750966 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:38:12.809931 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:12.809901 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:38:13.057412 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:13.057365 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" event={"ID":"8b28f8dc-5933-45b9-ad31-94a3492ee25e","Type":"ContainerStarted","Data":"c467b66e791e1d750feb29ef5ec010327812cfae683d08c5c298b0e40feb1f6b"} Jul 16 22:38:13.209441 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:13.209408 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" Jul 16 22:38:13.209719 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:13.209700 2585 patch_prober.go:28] interesting pod/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 16 22:38:13.209768 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:13.209741 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" podUID="8b28f8dc-5933-45b9-ad31-94a3492ee25e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 16 22:38:14.209314 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:14.209282 2585 patch_prober.go:28] interesting pod/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 16 22:38:14.209793 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:14.209762 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" podUID="8b28f8dc-5933-45b9-ad31-94a3492ee25e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 16 22:38:15.209815 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:15.209777 2585 patch_prober.go:28] interesting pod/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 16 22:38:15.210206 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:15.209839 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" podUID="8b28f8dc-5933-45b9-ad31-94a3492ee25e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 16 22:38:15.999388 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:15.999346 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:38:16.208962 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:16.208921 2585 patch_prober.go:28] interesting pod/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 16 22:38:16.209136 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:16.208989 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" podUID="8b28f8dc-5933-45b9-ad31-94a3492ee25e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 16 22:38:17.212755 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:17.212722 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" Jul 16 22:38:17.213144 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:17.212996 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" Jul 16 22:38:17.213727 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:17.213706 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" Jul 16 22:38:18.089583 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:18.089553 2585 generic.go:356] "Generic (PLEG): container finished" podID="bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f" containerID="795b1754ac421823b8c878b7be7d63e8b7db7125a9af3f173c9f7c76bc21bed9" exitCode=0 Jul 16 22:38:18.089754 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:18.089624 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" event={"ID":"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f","Type":"ContainerDied","Data":"795b1754ac421823b8c878b7be7d63e8b7db7125a9af3f173c9f7c76bc21bed9"} Jul 16 22:38:18.090108 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:18.090088 2585 scope.go:117] "RemoveContainer" containerID="795b1754ac421823b8c878b7be7d63e8b7db7125a9af3f173c9f7c76bc21bed9" Jul 16 22:38:18.091087 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:18.091065 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 22:38:18.091153 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:18.091133 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 22:38:18.091194 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:18.091166 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 22:38:19.097840 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:19.097803 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" event={"ID":"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f","Type":"ContainerStarted","Data":"03b662f41a93f5358d3d2c1dc898211a39946bf198b174fadc60f6644e819727"} Jul 16 22:38:19.354340 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:19.354235 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" Jul 16 22:38:19.355535 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:19.355509 2585 patch_prober.go:28] interesting pod/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 16 22:38:19.355659 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:19.355574 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" podUID="bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 16 22:38:19.540411 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:19.540368 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:38:19.643967 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:19.643845 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:38:19.999321 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:19.999239 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" Jul 16 22:38:20.354737 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:20.354701 2585 patch_prober.go:28] interesting pod/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 16 22:38:20.355140 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:20.354758 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" podUID="bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 16 22:38:21.354965 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:21.354928 2585 patch_prober.go:28] interesting pod/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 16 22:38:21.355395 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:21.355001 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" podUID="bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 16 22:38:22.354984 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:22.354944 2585 patch_prober.go:28] interesting pod/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 16 22:38:22.355403 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:22.355020 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" podUID="bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 16 22:38:23.364445 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:23.364414 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" Jul 16 22:38:23.365557 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:23.365537 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" Jul 16 22:38:27.446365 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:27.446290 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:38:28.287349 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:28.287317 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:38:28.387881 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:28.387848 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:38:30.600396 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:30.600364 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:38:30.697948 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:30.697916 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:38:31.034298 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:31.034201 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:38:47.005178 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:47.005144 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-7d2f73/auth-test-model-7d2f73-kserve-75469664f5-krq6q"] Jul 16 22:38:47.005718 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:47.005670 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49dc4ef4-bde7-4259-b4d7-13ca3950f05e" containerName="istio-proxy" Jul 16 22:38:47.005718 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:47.005686 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="49dc4ef4-bde7-4259-b4d7-13ca3950f05e" containerName="istio-proxy" Jul 16 22:38:47.008668 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:47.008651 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7d2f73/auth-test-model-7d2f73-kserve-75469664f5-krq6q" Jul 16 22:38:47.011381 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:47.011360 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-7d2f73\"/\"default-dockercfg-ftw2r\"" Jul 16 22:38:47.012268 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:47.012251 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-7d2f73\"/\"kube-root-ca.crt\"" Jul 16 22:38:47.012368 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:47.012308 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-7d2f73\"/\"auth-test-model-7d2f73-kserve-self-signed-certs\"" Jul 16 22:38:47.012368 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:47.012336 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-7d2f73\"/\"openshift-service-ca.crt\"" Jul 16 22:38:47.027492 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:47.027469 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9afacc3e-1799-46f7-8ce7-ad2af4081c06-dshm\") pod \"auth-test-model-7d2f73-kserve-75469664f5-krq6q\" (UID: \"9afacc3e-1799-46f7-8ce7-ad2af4081c06\") " pod="ai-tenant-e2e-shared-b-7d2f73/auth-test-model-7d2f73-kserve-75469664f5-krq6q" Jul 16 22:38:47.027586 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:47.027493 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9afacc3e-1799-46f7-8ce7-ad2af4081c06-tmp-dir\") pod \"auth-test-model-7d2f73-kserve-75469664f5-krq6q\" (UID: \"9afacc3e-1799-46f7-8ce7-ad2af4081c06\") " pod="ai-tenant-e2e-shared-b-7d2f73/auth-test-model-7d2f73-kserve-75469664f5-krq6q" Jul 16 22:38:47.027586 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:47.027523 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9afacc3e-1799-46f7-8ce7-ad2af4081c06-tls-certs\") pod \"auth-test-model-7d2f73-kserve-75469664f5-krq6q\" (UID: \"9afacc3e-1799-46f7-8ce7-ad2af4081c06\") " pod="ai-tenant-e2e-shared-b-7d2f73/auth-test-model-7d2f73-kserve-75469664f5-krq6q" Jul 16 22:38:47.027586 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:47.027538 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d2mk\" (UniqueName: \"kubernetes.io/projected/9afacc3e-1799-46f7-8ce7-ad2af4081c06-kube-api-access-9d2mk\") pod \"auth-test-model-7d2f73-kserve-75469664f5-krq6q\" (UID: \"9afacc3e-1799-46f7-8ce7-ad2af4081c06\") " pod="ai-tenant-e2e-shared-b-7d2f73/auth-test-model-7d2f73-kserve-75469664f5-krq6q" Jul 16 22:38:47.027694 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:47.027601 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9afacc3e-1799-46f7-8ce7-ad2af4081c06-model-cache\") pod \"auth-test-model-7d2f73-kserve-75469664f5-krq6q\" (UID: \"9afacc3e-1799-46f7-8ce7-ad2af4081c06\") " pod="ai-tenant-e2e-shared-b-7d2f73/auth-test-model-7d2f73-kserve-75469664f5-krq6q" Jul 16 22:38:47.027694 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:47.027627 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9afacc3e-1799-46f7-8ce7-ad2af4081c06-home\") pod \"auth-test-model-7d2f73-kserve-75469664f5-krq6q\" (UID: \"9afacc3e-1799-46f7-8ce7-ad2af4081c06\") " pod="ai-tenant-e2e-shared-b-7d2f73/auth-test-model-7d2f73-kserve-75469664f5-krq6q" Jul 16 22:38:47.044966 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:47.044946 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-7d2f73/auth-test-model-7d2f73-kserve-75469664f5-krq6q"] Jul 16 22:38:47.101912 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:47.101884 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:38:47.128244 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:47.128203 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9afacc3e-1799-46f7-8ce7-ad2af4081c06-dshm\") pod \"auth-test-model-7d2f73-kserve-75469664f5-krq6q\" (UID: \"9afacc3e-1799-46f7-8ce7-ad2af4081c06\") " pod="ai-tenant-e2e-shared-b-7d2f73/auth-test-model-7d2f73-kserve-75469664f5-krq6q" Jul 16 22:38:47.128381 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:47.128252 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9afacc3e-1799-46f7-8ce7-ad2af4081c06-tmp-dir\") pod \"auth-test-model-7d2f73-kserve-75469664f5-krq6q\" (UID: \"9afacc3e-1799-46f7-8ce7-ad2af4081c06\") " pod="ai-tenant-e2e-shared-b-7d2f73/auth-test-model-7d2f73-kserve-75469664f5-krq6q" Jul 16 22:38:47.128381 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:47.128300 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9afacc3e-1799-46f7-8ce7-ad2af4081c06-tls-certs\") pod \"auth-test-model-7d2f73-kserve-75469664f5-krq6q\" (UID: \"9afacc3e-1799-46f7-8ce7-ad2af4081c06\") " pod="ai-tenant-e2e-shared-b-7d2f73/auth-test-model-7d2f73-kserve-75469664f5-krq6q" Jul 16 22:38:47.128381 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:47.128324 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d2mk\" (UniqueName: \"kubernetes.io/projected/9afacc3e-1799-46f7-8ce7-ad2af4081c06-kube-api-access-9d2mk\") pod \"auth-test-model-7d2f73-kserve-75469664f5-krq6q\" (UID: \"9afacc3e-1799-46f7-8ce7-ad2af4081c06\") " pod="ai-tenant-e2e-shared-b-7d2f73/auth-test-model-7d2f73-kserve-75469664f5-krq6q" Jul 16 22:38:47.128381 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:47.128376 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9afacc3e-1799-46f7-8ce7-ad2af4081c06-model-cache\") pod \"auth-test-model-7d2f73-kserve-75469664f5-krq6q\" (UID: \"9afacc3e-1799-46f7-8ce7-ad2af4081c06\") " pod="ai-tenant-e2e-shared-b-7d2f73/auth-test-model-7d2f73-kserve-75469664f5-krq6q" Jul 16 22:38:47.128597 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:47.128409 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9afacc3e-1799-46f7-8ce7-ad2af4081c06-home\") pod \"auth-test-model-7d2f73-kserve-75469664f5-krq6q\" (UID: \"9afacc3e-1799-46f7-8ce7-ad2af4081c06\") " pod="ai-tenant-e2e-shared-b-7d2f73/auth-test-model-7d2f73-kserve-75469664f5-krq6q" Jul 16 22:38:47.128700 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:47.128674 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9afacc3e-1799-46f7-8ce7-ad2af4081c06-tmp-dir\") pod \"auth-test-model-7d2f73-kserve-75469664f5-krq6q\" (UID: \"9afacc3e-1799-46f7-8ce7-ad2af4081c06\") " pod="ai-tenant-e2e-shared-b-7d2f73/auth-test-model-7d2f73-kserve-75469664f5-krq6q" Jul 16 22:38:47.128767 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:47.128730 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9afacc3e-1799-46f7-8ce7-ad2af4081c06-model-cache\") pod \"auth-test-model-7d2f73-kserve-75469664f5-krq6q\" (UID: \"9afacc3e-1799-46f7-8ce7-ad2af4081c06\") " pod="ai-tenant-e2e-shared-b-7d2f73/auth-test-model-7d2f73-kserve-75469664f5-krq6q" Jul 16 22:38:47.128827 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:47.128809 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9afacc3e-1799-46f7-8ce7-ad2af4081c06-home\") pod \"auth-test-model-7d2f73-kserve-75469664f5-krq6q\" (UID: \"9afacc3e-1799-46f7-8ce7-ad2af4081c06\") " pod="ai-tenant-e2e-shared-b-7d2f73/auth-test-model-7d2f73-kserve-75469664f5-krq6q" Jul 16 22:38:47.130434 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:47.130418 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9afacc3e-1799-46f7-8ce7-ad2af4081c06-dshm\") pod \"auth-test-model-7d2f73-kserve-75469664f5-krq6q\" (UID: \"9afacc3e-1799-46f7-8ce7-ad2af4081c06\") " pod="ai-tenant-e2e-shared-b-7d2f73/auth-test-model-7d2f73-kserve-75469664f5-krq6q" Jul 16 22:38:47.130804 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:47.130784 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9afacc3e-1799-46f7-8ce7-ad2af4081c06-tls-certs\") pod \"auth-test-model-7d2f73-kserve-75469664f5-krq6q\" (UID: \"9afacc3e-1799-46f7-8ce7-ad2af4081c06\") " pod="ai-tenant-e2e-shared-b-7d2f73/auth-test-model-7d2f73-kserve-75469664f5-krq6q" Jul 16 22:38:47.142828 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:47.142802 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d2mk\" (UniqueName: \"kubernetes.io/projected/9afacc3e-1799-46f7-8ce7-ad2af4081c06-kube-api-access-9d2mk\") pod \"auth-test-model-7d2f73-kserve-75469664f5-krq6q\" (UID: \"9afacc3e-1799-46f7-8ce7-ad2af4081c06\") " pod="ai-tenant-e2e-shared-b-7d2f73/auth-test-model-7d2f73-kserve-75469664f5-krq6q" Jul 16 22:38:47.317984 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:47.317959 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7d2f73/auth-test-model-7d2f73-kserve-75469664f5-krq6q" Jul 16 22:38:47.467526 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:47.467495 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-7d2f73/auth-test-model-7d2f73-kserve-75469664f5-krq6q"] Jul 16 22:38:47.469089 ip-10-0-128-255 kubenswrapper[2585]: W0716 22:38:47.469058 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9afacc3e_1799_46f7_8ce7_ad2af4081c06.slice/crio-24ada44e805d3949fe42fc69eda60d4ab7fbe091db84eba91d0616467501942b WatchSource:0}: Error finding container 24ada44e805d3949fe42fc69eda60d4ab7fbe091db84eba91d0616467501942b: Status 404 returned error can't find the container with id 24ada44e805d3949fe42fc69eda60d4ab7fbe091db84eba91d0616467501942b Jul 16 22:38:48.260751 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:48.260713 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7d2f73/auth-test-model-7d2f73-kserve-75469664f5-krq6q" event={"ID":"9afacc3e-1799-46f7-8ce7-ad2af4081c06","Type":"ContainerStarted","Data":"8a5d850335c1ef681c8d5539ec21334a13618b5abe1d34d7ffc2abb5be7a75e3"} Jul 16 22:38:48.261155 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:48.260760 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7d2f73/auth-test-model-7d2f73-kserve-75469664f5-krq6q" event={"ID":"9afacc3e-1799-46f7-8ce7-ad2af4081c06","Type":"ContainerStarted","Data":"24ada44e805d3949fe42fc69eda60d4ab7fbe091db84eba91d0616467501942b"} Jul 16 22:38:48.295291 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:48.295238 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-7d2f73/auth-test-model-7d2f73-kserve-75469664f5-krq6q" podStartSLOduration=2.2952056499999998 podStartE2EDuration="2.29520565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:38:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:38:48.294627299 +0000 UTC m=+3076.917070253" watchObservedRunningTime="2026-07-16 22:38:48.29520565 +0000 UTC m=+3076.917648605" Jul 16 22:38:51.306943 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:51.306911 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:38:57.318752 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:57.318716 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-7d2f73/auth-test-model-7d2f73-kserve-75469664f5-krq6q" Jul 16 22:38:57.319245 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:57.318772 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-7d2f73/auth-test-model-7d2f73-kserve-75469664f5-krq6q" Jul 16 22:38:57.331491 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:57.331472 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-7d2f73/auth-test-model-7d2f73-kserve-75469664f5-krq6q" Jul 16 22:38:58.322021 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:38:58.321990 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-7d2f73/auth-test-model-7d2f73-kserve-75469664f5-krq6q" Jul 16 22:39:07.455863 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:39:07.455824 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:39:11.030279 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:39:11.030239 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:39:25.906191 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:39:25.906157 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:39:43.810553 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:39:43.810517 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:40:01.433079 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:40:01.432990 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:40:16.305373 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:40:16.305331 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:40:30.604770 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:40:30.604733 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:40:48.403986 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:40:48.403950 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:40:52.307687 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:40:52.307630 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:40:53.100453 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:40:53.100423 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-7d2f73/auth-test-model-7d2f73-kserve-75469664f5-krq6q"] Jul 16 22:40:53.133020 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:40:53.132984 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:41:05.502840 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:05.502805 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:41:08.120213 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:08.120154 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-7d2f73/auth-test-model-7d2f73-kserve-75469664f5-krq6q" podUID="9afacc3e-1799-46f7-8ce7-ad2af4081c06" containerName="main" containerID="cri-o://8a5d850335c1ef681c8d5539ec21334a13618b5abe1d34d7ffc2abb5be7a75e3" gracePeriod=45 Jul 16 22:41:08.371247 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:08.371160 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7d2f73/auth-test-model-7d2f73-kserve-75469664f5-krq6q" Jul 16 22:41:08.487402 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:08.487354 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9afacc3e-1799-46f7-8ce7-ad2af4081c06-home\") pod \"9afacc3e-1799-46f7-8ce7-ad2af4081c06\" (UID: \"9afacc3e-1799-46f7-8ce7-ad2af4081c06\") " Jul 16 22:41:08.487402 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:08.487414 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9afacc3e-1799-46f7-8ce7-ad2af4081c06-model-cache\") pod \"9afacc3e-1799-46f7-8ce7-ad2af4081c06\" (UID: \"9afacc3e-1799-46f7-8ce7-ad2af4081c06\") " Jul 16 22:41:08.487714 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:08.487442 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9d2mk\" (UniqueName: \"kubernetes.io/projected/9afacc3e-1799-46f7-8ce7-ad2af4081c06-kube-api-access-9d2mk\") pod \"9afacc3e-1799-46f7-8ce7-ad2af4081c06\" (UID: \"9afacc3e-1799-46f7-8ce7-ad2af4081c06\") " Jul 16 22:41:08.487714 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:08.487460 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9afacc3e-1799-46f7-8ce7-ad2af4081c06-tls-certs\") pod \"9afacc3e-1799-46f7-8ce7-ad2af4081c06\" (UID: \"9afacc3e-1799-46f7-8ce7-ad2af4081c06\") " Jul 16 22:41:08.487714 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:08.487500 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9afacc3e-1799-46f7-8ce7-ad2af4081c06-dshm\") pod \"9afacc3e-1799-46f7-8ce7-ad2af4081c06\" (UID: \"9afacc3e-1799-46f7-8ce7-ad2af4081c06\") " Jul 16 22:41:08.487714 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:08.487542 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9afacc3e-1799-46f7-8ce7-ad2af4081c06-tmp-dir\") pod \"9afacc3e-1799-46f7-8ce7-ad2af4081c06\" (UID: \"9afacc3e-1799-46f7-8ce7-ad2af4081c06\") " Jul 16 22:41:08.487714 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:08.487699 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9afacc3e-1799-46f7-8ce7-ad2af4081c06-home" (OuterVolumeSpecName: "home") pod "9afacc3e-1799-46f7-8ce7-ad2af4081c06" (UID: "9afacc3e-1799-46f7-8ce7-ad2af4081c06"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:41:08.487985 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:08.487764 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9afacc3e-1799-46f7-8ce7-ad2af4081c06-model-cache" (OuterVolumeSpecName: "model-cache") pod "9afacc3e-1799-46f7-8ce7-ad2af4081c06" (UID: "9afacc3e-1799-46f7-8ce7-ad2af4081c06"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:41:08.487985 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:08.487865 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9afacc3e-1799-46f7-8ce7-ad2af4081c06-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9afacc3e-1799-46f7-8ce7-ad2af4081c06" (UID: "9afacc3e-1799-46f7-8ce7-ad2af4081c06"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:41:08.487985 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:08.487902 2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9afacc3e-1799-46f7-8ce7-ad2af4081c06-home\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:41:08.487985 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:08.487920 2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9afacc3e-1799-46f7-8ce7-ad2af4081c06-model-cache\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:41:08.489829 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:08.489808 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9afacc3e-1799-46f7-8ce7-ad2af4081c06-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9afacc3e-1799-46f7-8ce7-ad2af4081c06" (UID: "9afacc3e-1799-46f7-8ce7-ad2af4081c06"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 22:41:08.489829 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:08.489823 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9afacc3e-1799-46f7-8ce7-ad2af4081c06-kube-api-access-9d2mk" (OuterVolumeSpecName: "kube-api-access-9d2mk") pod "9afacc3e-1799-46f7-8ce7-ad2af4081c06" (UID: "9afacc3e-1799-46f7-8ce7-ad2af4081c06"). InnerVolumeSpecName "kube-api-access-9d2mk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 22:41:08.489948 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:08.489870 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9afacc3e-1799-46f7-8ce7-ad2af4081c06-dshm" (OuterVolumeSpecName: "dshm") pod "9afacc3e-1799-46f7-8ce7-ad2af4081c06" (UID: "9afacc3e-1799-46f7-8ce7-ad2af4081c06"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:41:08.588705 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:08.588665 2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9afacc3e-1799-46f7-8ce7-ad2af4081c06-dshm\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:41:08.588705 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:08.588699 2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9afacc3e-1799-46f7-8ce7-ad2af4081c06-tmp-dir\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:41:08.588705 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:08.588711 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d2mk\" (UniqueName: \"kubernetes.io/projected/9afacc3e-1799-46f7-8ce7-ad2af4081c06-kube-api-access-9d2mk\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:41:08.588944 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:08.588719 2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9afacc3e-1799-46f7-8ce7-ad2af4081c06-tls-certs\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:41:08.977847 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:08.977812 2585 generic.go:356] "Generic (PLEG): container finished" podID="9afacc3e-1799-46f7-8ce7-ad2af4081c06" containerID="8a5d850335c1ef681c8d5539ec21334a13618b5abe1d34d7ffc2abb5be7a75e3" exitCode=0 Jul 16 22:41:08.978048 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:08.977852 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7d2f73/auth-test-model-7d2f73-kserve-75469664f5-krq6q" Jul 16 22:41:08.978048 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:08.977905 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7d2f73/auth-test-model-7d2f73-kserve-75469664f5-krq6q" event={"ID":"9afacc3e-1799-46f7-8ce7-ad2af4081c06","Type":"ContainerDied","Data":"8a5d850335c1ef681c8d5539ec21334a13618b5abe1d34d7ffc2abb5be7a75e3"} Jul 16 22:41:08.978048 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:08.977949 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7d2f73/auth-test-model-7d2f73-kserve-75469664f5-krq6q" event={"ID":"9afacc3e-1799-46f7-8ce7-ad2af4081c06","Type":"ContainerDied","Data":"24ada44e805d3949fe42fc69eda60d4ab7fbe091db84eba91d0616467501942b"} Jul 16 22:41:08.978048 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:08.977972 2585 scope.go:117] "RemoveContainer" containerID="8a5d850335c1ef681c8d5539ec21334a13618b5abe1d34d7ffc2abb5be7a75e3" Jul 16 22:41:08.987187 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:08.987170 2585 scope.go:117] "RemoveContainer" containerID="8a5d850335c1ef681c8d5539ec21334a13618b5abe1d34d7ffc2abb5be7a75e3" Jul 16 22:41:08.987442 ip-10-0-128-255 kubenswrapper[2585]: E0716 22:41:08.987420 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a5d850335c1ef681c8d5539ec21334a13618b5abe1d34d7ffc2abb5be7a75e3\": container with ID starting with 8a5d850335c1ef681c8d5539ec21334a13618b5abe1d34d7ffc2abb5be7a75e3 not found: ID does not exist" containerID="8a5d850335c1ef681c8d5539ec21334a13618b5abe1d34d7ffc2abb5be7a75e3" Jul 16 22:41:08.987505 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:08.987452 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a5d850335c1ef681c8d5539ec21334a13618b5abe1d34d7ffc2abb5be7a75e3"} err="failed to get container status \"8a5d850335c1ef681c8d5539ec21334a13618b5abe1d34d7ffc2abb5be7a75e3\": rpc error: code = NotFound desc = could not find container \"8a5d850335c1ef681c8d5539ec21334a13618b5abe1d34d7ffc2abb5be7a75e3\": container with ID starting with 8a5d850335c1ef681c8d5539ec21334a13618b5abe1d34d7ffc2abb5be7a75e3 not found: ID does not exist" Jul 16 22:41:09.000299 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:09.000273 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-7d2f73/auth-test-model-7d2f73-kserve-75469664f5-krq6q"] Jul 16 22:41:09.007943 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:09.007921 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-7d2f73/auth-test-model-7d2f73-kserve-75469664f5-krq6q"] Jul 16 22:41:09.935042 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:09.935006 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9afacc3e-1799-46f7-8ce7-ad2af4081c06" path="/var/lib/kubelet/pods/9afacc3e-1799-46f7-8ce7-ad2af4081c06/volumes" Jul 16 22:41:13.311643 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:13.311598 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-7d2f73/auth-test-model-7d2f73-kserve-75469664f5-krq6q" podUID="9afacc3e-1799-46f7-8ce7-ad2af4081c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Jul 16 22:41:14.423927 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:14.423891 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:41:18.851870 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:18.851829 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:41:32.783666 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:32.783632 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-7d2f73/sub-test-model-7d2f73-kserve-778587d545-vqp6x"] Jul 16 22:41:32.784078 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:32.784027 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9afacc3e-1799-46f7-8ce7-ad2af4081c06" containerName="main" Jul 16 22:41:32.784078 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:32.784040 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="9afacc3e-1799-46f7-8ce7-ad2af4081c06" containerName="main" Jul 16 22:41:32.784152 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:32.784107 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="9afacc3e-1799-46f7-8ce7-ad2af4081c06" containerName="main" Jul 16 22:41:32.788500 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:32.788480 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7d2f73/sub-test-model-7d2f73-kserve-778587d545-vqp6x" Jul 16 22:41:32.792063 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:32.792039 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-7d2f73\"/\"kube-root-ca.crt\"" Jul 16 22:41:32.792195 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:32.792113 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-7d2f73\"/\"default-dockercfg-ftw2r\"" Jul 16 22:41:32.792195 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:32.792139 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-7d2f73\"/\"openshift-service-ca.crt\"" Jul 16 22:41:32.792195 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:32.792050 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-7d2f73\"/\"sub-test-model-7d2f73-kserve-self-signed-certs\"" Jul 16 22:41:32.798994 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:32.798969 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-7d2f73/sub-test-model-7d2f73-kserve-778587d545-vqp6x"] Jul 16 22:41:32.824376 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:32.824344 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:41:32.920752 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:32.920716 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/779d3319-bb85-4d62-a1eb-3c27d4339c88-dshm\") pod \"sub-test-model-7d2f73-kserve-778587d545-vqp6x\" (UID: \"779d3319-bb85-4d62-a1eb-3c27d4339c88\") " pod="ai-tenant-e2e-shared-b-7d2f73/sub-test-model-7d2f73-kserve-778587d545-vqp6x" Jul 16 22:41:32.920932 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:32.920758 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/779d3319-bb85-4d62-a1eb-3c27d4339c88-tmp-dir\") pod \"sub-test-model-7d2f73-kserve-778587d545-vqp6x\" (UID: \"779d3319-bb85-4d62-a1eb-3c27d4339c88\") " pod="ai-tenant-e2e-shared-b-7d2f73/sub-test-model-7d2f73-kserve-778587d545-vqp6x" Jul 16 22:41:32.920932 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:32.920785 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrmxl\" (UniqueName: \"kubernetes.io/projected/779d3319-bb85-4d62-a1eb-3c27d4339c88-kube-api-access-lrmxl\") pod \"sub-test-model-7d2f73-kserve-778587d545-vqp6x\" (UID: \"779d3319-bb85-4d62-a1eb-3c27d4339c88\") " pod="ai-tenant-e2e-shared-b-7d2f73/sub-test-model-7d2f73-kserve-778587d545-vqp6x" Jul 16 22:41:32.920932 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:32.920896 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/779d3319-bb85-4d62-a1eb-3c27d4339c88-model-cache\") pod \"sub-test-model-7d2f73-kserve-778587d545-vqp6x\" (UID: \"779d3319-bb85-4d62-a1eb-3c27d4339c88\") " pod="ai-tenant-e2e-shared-b-7d2f73/sub-test-model-7d2f73-kserve-778587d545-vqp6x" Jul 16 22:41:32.921071 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:32.920943 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/779d3319-bb85-4d62-a1eb-3c27d4339c88-tls-certs\") pod \"sub-test-model-7d2f73-kserve-778587d545-vqp6x\" (UID: \"779d3319-bb85-4d62-a1eb-3c27d4339c88\") " pod="ai-tenant-e2e-shared-b-7d2f73/sub-test-model-7d2f73-kserve-778587d545-vqp6x" Jul 16 22:41:32.921071 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:32.920999 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/779d3319-bb85-4d62-a1eb-3c27d4339c88-home\") pod \"sub-test-model-7d2f73-kserve-778587d545-vqp6x\" (UID: \"779d3319-bb85-4d62-a1eb-3c27d4339c88\") " pod="ai-tenant-e2e-shared-b-7d2f73/sub-test-model-7d2f73-kserve-778587d545-vqp6x" Jul 16 22:41:33.021598 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:33.021564 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/779d3319-bb85-4d62-a1eb-3c27d4339c88-tmp-dir\") pod \"sub-test-model-7d2f73-kserve-778587d545-vqp6x\" (UID: \"779d3319-bb85-4d62-a1eb-3c27d4339c88\") " pod="ai-tenant-e2e-shared-b-7d2f73/sub-test-model-7d2f73-kserve-778587d545-vqp6x" Jul 16 22:41:33.021791 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:33.021611 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrmxl\" (UniqueName: \"kubernetes.io/projected/779d3319-bb85-4d62-a1eb-3c27d4339c88-kube-api-access-lrmxl\") pod \"sub-test-model-7d2f73-kserve-778587d545-vqp6x\" (UID: \"779d3319-bb85-4d62-a1eb-3c27d4339c88\") " pod="ai-tenant-e2e-shared-b-7d2f73/sub-test-model-7d2f73-kserve-778587d545-vqp6x" Jul 16 22:41:33.021791 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:33.021655 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/779d3319-bb85-4d62-a1eb-3c27d4339c88-model-cache\") pod \"sub-test-model-7d2f73-kserve-778587d545-vqp6x\" (UID: \"779d3319-bb85-4d62-a1eb-3c27d4339c88\") " pod="ai-tenant-e2e-shared-b-7d2f73/sub-test-model-7d2f73-kserve-778587d545-vqp6x" Jul 16 22:41:33.021791 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:33.021685 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/779d3319-bb85-4d62-a1eb-3c27d4339c88-tls-certs\") pod \"sub-test-model-7d2f73-kserve-778587d545-vqp6x\" (UID: \"779d3319-bb85-4d62-a1eb-3c27d4339c88\") " pod="ai-tenant-e2e-shared-b-7d2f73/sub-test-model-7d2f73-kserve-778587d545-vqp6x" Jul 16 22:41:33.021791 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:33.021709 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/779d3319-bb85-4d62-a1eb-3c27d4339c88-home\") pod \"sub-test-model-7d2f73-kserve-778587d545-vqp6x\" (UID: \"779d3319-bb85-4d62-a1eb-3c27d4339c88\") " pod="ai-tenant-e2e-shared-b-7d2f73/sub-test-model-7d2f73-kserve-778587d545-vqp6x" Jul 16 22:41:33.021791 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:33.021778 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/779d3319-bb85-4d62-a1eb-3c27d4339c88-dshm\") pod \"sub-test-model-7d2f73-kserve-778587d545-vqp6x\" (UID: \"779d3319-bb85-4d62-a1eb-3c27d4339c88\") " pod="ai-tenant-e2e-shared-b-7d2f73/sub-test-model-7d2f73-kserve-778587d545-vqp6x" Jul 16 22:41:33.022057 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:33.022026 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/779d3319-bb85-4d62-a1eb-3c27d4339c88-tmp-dir\") pod \"sub-test-model-7d2f73-kserve-778587d545-vqp6x\" (UID: \"779d3319-bb85-4d62-a1eb-3c27d4339c88\") " pod="ai-tenant-e2e-shared-b-7d2f73/sub-test-model-7d2f73-kserve-778587d545-vqp6x" Jul 16 22:41:33.022116 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:33.022065 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/779d3319-bb85-4d62-a1eb-3c27d4339c88-model-cache\") pod \"sub-test-model-7d2f73-kserve-778587d545-vqp6x\" (UID: \"779d3319-bb85-4d62-a1eb-3c27d4339c88\") " pod="ai-tenant-e2e-shared-b-7d2f73/sub-test-model-7d2f73-kserve-778587d545-vqp6x" Jul 16 22:41:33.022116 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:33.022081 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/779d3319-bb85-4d62-a1eb-3c27d4339c88-home\") pod \"sub-test-model-7d2f73-kserve-778587d545-vqp6x\" (UID: \"779d3319-bb85-4d62-a1eb-3c27d4339c88\") " pod="ai-tenant-e2e-shared-b-7d2f73/sub-test-model-7d2f73-kserve-778587d545-vqp6x" Jul 16 22:41:33.024084 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:33.024058 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/779d3319-bb85-4d62-a1eb-3c27d4339c88-dshm\") pod \"sub-test-model-7d2f73-kserve-778587d545-vqp6x\" (UID: \"779d3319-bb85-4d62-a1eb-3c27d4339c88\") " pod="ai-tenant-e2e-shared-b-7d2f73/sub-test-model-7d2f73-kserve-778587d545-vqp6x" Jul 16 22:41:33.024329 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:33.024309 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/779d3319-bb85-4d62-a1eb-3c27d4339c88-tls-certs\") pod \"sub-test-model-7d2f73-kserve-778587d545-vqp6x\" (UID: \"779d3319-bb85-4d62-a1eb-3c27d4339c88\") " pod="ai-tenant-e2e-shared-b-7d2f73/sub-test-model-7d2f73-kserve-778587d545-vqp6x" Jul 16 22:41:33.033202 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:33.033172 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrmxl\" (UniqueName: \"kubernetes.io/projected/779d3319-bb85-4d62-a1eb-3c27d4339c88-kube-api-access-lrmxl\") pod \"sub-test-model-7d2f73-kserve-778587d545-vqp6x\" (UID: \"779d3319-bb85-4d62-a1eb-3c27d4339c88\") " pod="ai-tenant-e2e-shared-b-7d2f73/sub-test-model-7d2f73-kserve-778587d545-vqp6x" Jul 16 22:41:33.098743 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:33.098714 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7d2f73/sub-test-model-7d2f73-kserve-778587d545-vqp6x" Jul 16 22:41:33.237269 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:33.237244 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-7d2f73/sub-test-model-7d2f73-kserve-778587d545-vqp6x"] Jul 16 22:41:33.238738 ip-10-0-128-255 kubenswrapper[2585]: W0716 22:41:33.238709 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod779d3319_bb85_4d62_a1eb_3c27d4339c88.slice/crio-fd45f661e87b2ccb6d2e4dc133365a57cd1e720e58455b28f7289db23aa9c324 WatchSource:0}: Error finding container fd45f661e87b2ccb6d2e4dc133365a57cd1e720e58455b28f7289db23aa9c324: Status 404 returned error can't find the container with id fd45f661e87b2ccb6d2e4dc133365a57cd1e720e58455b28f7289db23aa9c324 Jul 16 22:41:34.106634 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:34.106599 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7d2f73/sub-test-model-7d2f73-kserve-778587d545-vqp6x" event={"ID":"779d3319-bb85-4d62-a1eb-3c27d4339c88","Type":"ContainerStarted","Data":"ccb93f8931086bee5bf90d5f5b5f2d069ed9bd42cc43c1c2e3f0bb881656986a"} Jul 16 22:41:34.106634 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:34.106641 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7d2f73/sub-test-model-7d2f73-kserve-778587d545-vqp6x" event={"ID":"779d3319-bb85-4d62-a1eb-3c27d4339c88","Type":"ContainerStarted","Data":"fd45f661e87b2ccb6d2e4dc133365a57cd1e720e58455b28f7289db23aa9c324"} Jul 16 22:41:34.135660 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:34.135608 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-7d2f73/sub-test-model-7d2f73-kserve-778587d545-vqp6x" podStartSLOduration=2.135592999 podStartE2EDuration="2.135592999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:41:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:41:34.131763144 +0000 UTC m=+3242.754206098" watchObservedRunningTime="2026-07-16 22:41:34.135592999 +0000 UTC m=+3242.758035952" Jul 16 22:41:38.451628 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:38.451588 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:41:43.098929 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:43.098888 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-7d2f73/sub-test-model-7d2f73-kserve-778587d545-vqp6x" Jul 16 22:41:43.099355 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:43.098948 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-7d2f73/sub-test-model-7d2f73-kserve-778587d545-vqp6x" Jul 16 22:41:43.111311 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:43.111284 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-7d2f73/sub-test-model-7d2f73-kserve-778587d545-vqp6x" Jul 16 22:41:43.167445 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:43.167415 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-7d2f73/sub-test-model-7d2f73-kserve-778587d545-vqp6x" Jul 16 22:41:58.343816 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:41:58.343785 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:42:03.856569 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:03.856538 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:42:14.004426 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:14.004390 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:42:21.363595 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:21.363558 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:42:21.364964 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:21.364933 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-7d2f73/sub-test-model-7d2f73-kserve-778587d545-vqp6x"] Jul 16 22:42:21.463664 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:21.463633 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:42:26.206368 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:26.206328 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-1c9596/rate-test-model-1c9596-kserve-8c46746c9-9h9tm"] Jul 16 22:42:26.209870 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:26.209846 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1c9596/rate-test-model-1c9596-kserve-8c46746c9-9h9tm" Jul 16 22:42:26.212563 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:26.212540 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1c9596\"/\"kube-root-ca.crt\"" Jul 16 22:42:26.213626 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:26.213602 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1c9596\"/\"openshift-service-ca.crt\"" Jul 16 22:42:26.213755 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:26.213645 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1c9596\"/\"rate-test-model-1c9596-kserve-self-signed-certs\"" Jul 16 22:42:26.213755 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:26.213665 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1c9596\"/\"default-dockercfg-wxbsp\"" Jul 16 22:42:26.221878 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:26.221854 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e87ee267-229e-49f6-a835-ea50db8cb5ea-model-cache\") pod \"rate-test-model-1c9596-kserve-8c46746c9-9h9tm\" (UID: \"e87ee267-229e-49f6-a835-ea50db8cb5ea\") " pod="ai-tenant-e2e-shared-a-1c9596/rate-test-model-1c9596-kserve-8c46746c9-9h9tm" Jul 16 22:42:26.221985 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:26.221885 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e87ee267-229e-49f6-a835-ea50db8cb5ea-tls-certs\") pod \"rate-test-model-1c9596-kserve-8c46746c9-9h9tm\" (UID: \"e87ee267-229e-49f6-a835-ea50db8cb5ea\") " pod="ai-tenant-e2e-shared-a-1c9596/rate-test-model-1c9596-kserve-8c46746c9-9h9tm" Jul 16 22:42:26.221985 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:26.221964 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gq8c\" (UniqueName: \"kubernetes.io/projected/e87ee267-229e-49f6-a835-ea50db8cb5ea-kube-api-access-6gq8c\") pod \"rate-test-model-1c9596-kserve-8c46746c9-9h9tm\" (UID: \"e87ee267-229e-49f6-a835-ea50db8cb5ea\") " pod="ai-tenant-e2e-shared-a-1c9596/rate-test-model-1c9596-kserve-8c46746c9-9h9tm" Jul 16 22:42:26.222083 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:26.222004 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e87ee267-229e-49f6-a835-ea50db8cb5ea-tmp-dir\") pod \"rate-test-model-1c9596-kserve-8c46746c9-9h9tm\" (UID: \"e87ee267-229e-49f6-a835-ea50db8cb5ea\") " pod="ai-tenant-e2e-shared-a-1c9596/rate-test-model-1c9596-kserve-8c46746c9-9h9tm" Jul 16 22:42:26.222083 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:26.222039 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e87ee267-229e-49f6-a835-ea50db8cb5ea-home\") pod \"rate-test-model-1c9596-kserve-8c46746c9-9h9tm\" (UID: \"e87ee267-229e-49f6-a835-ea50db8cb5ea\") " pod="ai-tenant-e2e-shared-a-1c9596/rate-test-model-1c9596-kserve-8c46746c9-9h9tm" Jul 16 22:42:26.222083 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:26.222055 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e87ee267-229e-49f6-a835-ea50db8cb5ea-dshm\") pod \"rate-test-model-1c9596-kserve-8c46746c9-9h9tm\" (UID: \"e87ee267-229e-49f6-a835-ea50db8cb5ea\") " pod="ai-tenant-e2e-shared-a-1c9596/rate-test-model-1c9596-kserve-8c46746c9-9h9tm" Jul 16 22:42:26.229695 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:26.229670 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1c9596/rate-test-model-1c9596-kserve-8c46746c9-9h9tm"] Jul 16 22:42:26.249777 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:26.249746 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:42:26.323318 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:26.323283 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gq8c\" (UniqueName: \"kubernetes.io/projected/e87ee267-229e-49f6-a835-ea50db8cb5ea-kube-api-access-6gq8c\") pod \"rate-test-model-1c9596-kserve-8c46746c9-9h9tm\" (UID: \"e87ee267-229e-49f6-a835-ea50db8cb5ea\") " pod="ai-tenant-e2e-shared-a-1c9596/rate-test-model-1c9596-kserve-8c46746c9-9h9tm" Jul 16 22:42:26.323510 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:26.323334 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e87ee267-229e-49f6-a835-ea50db8cb5ea-tmp-dir\") pod \"rate-test-model-1c9596-kserve-8c46746c9-9h9tm\" (UID: \"e87ee267-229e-49f6-a835-ea50db8cb5ea\") " pod="ai-tenant-e2e-shared-a-1c9596/rate-test-model-1c9596-kserve-8c46746c9-9h9tm" Jul 16 22:42:26.323510 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:26.323371 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e87ee267-229e-49f6-a835-ea50db8cb5ea-home\") pod \"rate-test-model-1c9596-kserve-8c46746c9-9h9tm\" (UID: \"e87ee267-229e-49f6-a835-ea50db8cb5ea\") " pod="ai-tenant-e2e-shared-a-1c9596/rate-test-model-1c9596-kserve-8c46746c9-9h9tm" Jul 16 22:42:26.323510 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:26.323390 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e87ee267-229e-49f6-a835-ea50db8cb5ea-dshm\") pod \"rate-test-model-1c9596-kserve-8c46746c9-9h9tm\" (UID: \"e87ee267-229e-49f6-a835-ea50db8cb5ea\") " pod="ai-tenant-e2e-shared-a-1c9596/rate-test-model-1c9596-kserve-8c46746c9-9h9tm" Jul 16 22:42:26.323510 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:26.323423 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e87ee267-229e-49f6-a835-ea50db8cb5ea-model-cache\") pod \"rate-test-model-1c9596-kserve-8c46746c9-9h9tm\" (UID: \"e87ee267-229e-49f6-a835-ea50db8cb5ea\") " pod="ai-tenant-e2e-shared-a-1c9596/rate-test-model-1c9596-kserve-8c46746c9-9h9tm" Jul 16 22:42:26.323510 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:26.323446 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e87ee267-229e-49f6-a835-ea50db8cb5ea-tls-certs\") pod \"rate-test-model-1c9596-kserve-8c46746c9-9h9tm\" (UID: \"e87ee267-229e-49f6-a835-ea50db8cb5ea\") " pod="ai-tenant-e2e-shared-a-1c9596/rate-test-model-1c9596-kserve-8c46746c9-9h9tm" Jul 16 22:42:26.323827 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:26.323806 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e87ee267-229e-49f6-a835-ea50db8cb5ea-home\") pod \"rate-test-model-1c9596-kserve-8c46746c9-9h9tm\" (UID: \"e87ee267-229e-49f6-a835-ea50db8cb5ea\") " pod="ai-tenant-e2e-shared-a-1c9596/rate-test-model-1c9596-kserve-8c46746c9-9h9tm" Jul 16 22:42:26.323886 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:26.323844 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e87ee267-229e-49f6-a835-ea50db8cb5ea-tmp-dir\") pod \"rate-test-model-1c9596-kserve-8c46746c9-9h9tm\" (UID: \"e87ee267-229e-49f6-a835-ea50db8cb5ea\") " pod="ai-tenant-e2e-shared-a-1c9596/rate-test-model-1c9596-kserve-8c46746c9-9h9tm" Jul 16 22:42:26.323974 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:26.323953 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e87ee267-229e-49f6-a835-ea50db8cb5ea-model-cache\") pod \"rate-test-model-1c9596-kserve-8c46746c9-9h9tm\" (UID: \"e87ee267-229e-49f6-a835-ea50db8cb5ea\") " pod="ai-tenant-e2e-shared-a-1c9596/rate-test-model-1c9596-kserve-8c46746c9-9h9tm" Jul 16 22:42:26.325763 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:26.325738 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e87ee267-229e-49f6-a835-ea50db8cb5ea-dshm\") pod \"rate-test-model-1c9596-kserve-8c46746c9-9h9tm\" (UID: \"e87ee267-229e-49f6-a835-ea50db8cb5ea\") " pod="ai-tenant-e2e-shared-a-1c9596/rate-test-model-1c9596-kserve-8c46746c9-9h9tm" Jul 16 22:42:26.326048 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:26.326027 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e87ee267-229e-49f6-a835-ea50db8cb5ea-tls-certs\") pod \"rate-test-model-1c9596-kserve-8c46746c9-9h9tm\" (UID: \"e87ee267-229e-49f6-a835-ea50db8cb5ea\") " pod="ai-tenant-e2e-shared-a-1c9596/rate-test-model-1c9596-kserve-8c46746c9-9h9tm" Jul 16 22:42:26.332604 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:26.332582 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gq8c\" (UniqueName: \"kubernetes.io/projected/e87ee267-229e-49f6-a835-ea50db8cb5ea-kube-api-access-6gq8c\") pod \"rate-test-model-1c9596-kserve-8c46746c9-9h9tm\" (UID: \"e87ee267-229e-49f6-a835-ea50db8cb5ea\") " pod="ai-tenant-e2e-shared-a-1c9596/rate-test-model-1c9596-kserve-8c46746c9-9h9tm" Jul 16 22:42:26.520917 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:26.520813 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1c9596/rate-test-model-1c9596-kserve-8c46746c9-9h9tm" Jul 16 22:42:26.654472 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:26.654443 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1c9596/rate-test-model-1c9596-kserve-8c46746c9-9h9tm"] Jul 16 22:42:26.655833 ip-10-0-128-255 kubenswrapper[2585]: W0716 22:42:26.655807 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode87ee267_229e_49f6_a835_ea50db8cb5ea.slice/crio-016ae36a26ce69ee2046c1a1ea873076aa2abc48eb6ef4a844cb877dd80e3827 WatchSource:0}: Error finding container 016ae36a26ce69ee2046c1a1ea873076aa2abc48eb6ef4a844cb877dd80e3827: Status 404 returned error can't find the container with id 016ae36a26ce69ee2046c1a1ea873076aa2abc48eb6ef4a844cb877dd80e3827 Jul 16 22:42:27.385633 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:27.385589 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1c9596/rate-test-model-1c9596-kserve-8c46746c9-9h9tm" event={"ID":"e87ee267-229e-49f6-a835-ea50db8cb5ea","Type":"ContainerStarted","Data":"1ab362bb4707644073b3d87b3cb30dc24d9d4ece2970cb384e9547f3b588a5c2"} Jul 16 22:42:27.386124 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:27.385648 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1c9596/rate-test-model-1c9596-kserve-8c46746c9-9h9tm" event={"ID":"e87ee267-229e-49f6-a835-ea50db8cb5ea","Type":"ContainerStarted","Data":"016ae36a26ce69ee2046c1a1ea873076aa2abc48eb6ef4a844cb877dd80e3827"} Jul 16 22:42:27.410485 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:27.410427 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-1c9596/rate-test-model-1c9596-kserve-8c46746c9-9h9tm" podStartSLOduration=1.410410103 podStartE2EDuration="1.410410103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:42:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:42:27.409062245 +0000 UTC m=+3296.031505200" watchObservedRunningTime="2026-07-16 22:42:27.410410103 +0000 UTC m=+3296.032853056" Jul 16 22:42:33.063442 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:33.063399 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:42:36.384668 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:36.384615 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-7d2f73/sub-test-model-7d2f73-kserve-778587d545-vqp6x" podUID="779d3319-bb85-4d62-a1eb-3c27d4339c88" containerName="main" containerID="cri-o://ccb93f8931086bee5bf90d5f5b5f2d069ed9bd42cc43c1c2e3f0bb881656986a" gracePeriod=45 Jul 16 22:42:36.521063 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:36.521035 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-1c9596/rate-test-model-1c9596-kserve-8c46746c9-9h9tm" Jul 16 22:42:36.521209 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:36.521086 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-1c9596/rate-test-model-1c9596-kserve-8c46746c9-9h9tm" Jul 16 22:42:36.538601 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:36.538569 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-1c9596/rate-test-model-1c9596-kserve-8c46746c9-9h9tm" Jul 16 22:42:36.640110 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:36.640051 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7d2f73/sub-test-model-7d2f73-kserve-778587d545-vqp6x" Jul 16 22:42:36.723009 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:36.722975 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/779d3319-bb85-4d62-a1eb-3c27d4339c88-tmp-dir\") pod \"779d3319-bb85-4d62-a1eb-3c27d4339c88\" (UID: \"779d3319-bb85-4d62-a1eb-3c27d4339c88\") " Jul 16 22:42:36.723189 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:36.723057 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/779d3319-bb85-4d62-a1eb-3c27d4339c88-dshm\") pod \"779d3319-bb85-4d62-a1eb-3c27d4339c88\" (UID: \"779d3319-bb85-4d62-a1eb-3c27d4339c88\") " Jul 16 22:42:36.723189 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:36.723076 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/779d3319-bb85-4d62-a1eb-3c27d4339c88-tls-certs\") pod \"779d3319-bb85-4d62-a1eb-3c27d4339c88\" (UID: \"779d3319-bb85-4d62-a1eb-3c27d4339c88\") " Jul 16 22:42:36.723189 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:36.723105 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrmxl\" (UniqueName: \"kubernetes.io/projected/779d3319-bb85-4d62-a1eb-3c27d4339c88-kube-api-access-lrmxl\") pod \"779d3319-bb85-4d62-a1eb-3c27d4339c88\" (UID: \"779d3319-bb85-4d62-a1eb-3c27d4339c88\") " Jul 16 22:42:36.723189 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:36.723132 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/779d3319-bb85-4d62-a1eb-3c27d4339c88-model-cache\") pod \"779d3319-bb85-4d62-a1eb-3c27d4339c88\" (UID: \"779d3319-bb85-4d62-a1eb-3c27d4339c88\") " Jul 16 22:42:36.723189 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:36.723151 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/779d3319-bb85-4d62-a1eb-3c27d4339c88-home\") pod \"779d3319-bb85-4d62-a1eb-3c27d4339c88\" (UID: \"779d3319-bb85-4d62-a1eb-3c27d4339c88\") " Jul 16 22:42:36.723511 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:36.723303 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/779d3319-bb85-4d62-a1eb-3c27d4339c88-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "779d3319-bb85-4d62-a1eb-3c27d4339c88" (UID: "779d3319-bb85-4d62-a1eb-3c27d4339c88"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:42:36.723511 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:36.723432 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/779d3319-bb85-4d62-a1eb-3c27d4339c88-model-cache" (OuterVolumeSpecName: "model-cache") pod "779d3319-bb85-4d62-a1eb-3c27d4339c88" (UID: "779d3319-bb85-4d62-a1eb-3c27d4339c88"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:42:36.723511 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:36.723458 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/779d3319-bb85-4d62-a1eb-3c27d4339c88-home" (OuterVolumeSpecName: "home") pod "779d3319-bb85-4d62-a1eb-3c27d4339c88" (UID: "779d3319-bb85-4d62-a1eb-3c27d4339c88"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:42:36.723663 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:36.723543 2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/779d3319-bb85-4d62-a1eb-3c27d4339c88-tmp-dir\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:42:36.723663 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:36.723558 2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/779d3319-bb85-4d62-a1eb-3c27d4339c88-model-cache\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:42:36.723663 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:36.723566 2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/779d3319-bb85-4d62-a1eb-3c27d4339c88-home\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:42:36.725193 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:36.725163 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/779d3319-bb85-4d62-a1eb-3c27d4339c88-dshm" (OuterVolumeSpecName: "dshm") pod "779d3319-bb85-4d62-a1eb-3c27d4339c88" (UID: "779d3319-bb85-4d62-a1eb-3c27d4339c88"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:42:36.725353 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:36.725210 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/779d3319-bb85-4d62-a1eb-3c27d4339c88-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "779d3319-bb85-4d62-a1eb-3c27d4339c88" (UID: "779d3319-bb85-4d62-a1eb-3c27d4339c88"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 22:42:36.725353 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:36.725253 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/779d3319-bb85-4d62-a1eb-3c27d4339c88-kube-api-access-lrmxl" (OuterVolumeSpecName: "kube-api-access-lrmxl") pod "779d3319-bb85-4d62-a1eb-3c27d4339c88" (UID: "779d3319-bb85-4d62-a1eb-3c27d4339c88"). InnerVolumeSpecName "kube-api-access-lrmxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 22:42:36.824411 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:36.824380 2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/779d3319-bb85-4d62-a1eb-3c27d4339c88-dshm\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:42:36.824411 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:36.824405 2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/779d3319-bb85-4d62-a1eb-3c27d4339c88-tls-certs\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:42:36.824411 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:36.824416 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrmxl\" (UniqueName: \"kubernetes.io/projected/779d3319-bb85-4d62-a1eb-3c27d4339c88-kube-api-access-lrmxl\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:42:37.438130 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:37.438095 2585 generic.go:356] "Generic (PLEG): container finished" podID="779d3319-bb85-4d62-a1eb-3c27d4339c88" containerID="ccb93f8931086bee5bf90d5f5b5f2d069ed9bd42cc43c1c2e3f0bb881656986a" exitCode=0 Jul 16 22:42:37.438603 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:37.438143 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7d2f73/sub-test-model-7d2f73-kserve-778587d545-vqp6x" Jul 16 22:42:37.438603 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:37.438186 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7d2f73/sub-test-model-7d2f73-kserve-778587d545-vqp6x" event={"ID":"779d3319-bb85-4d62-a1eb-3c27d4339c88","Type":"ContainerDied","Data":"ccb93f8931086bee5bf90d5f5b5f2d069ed9bd42cc43c1c2e3f0bb881656986a"} Jul 16 22:42:37.438603 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:37.438255 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7d2f73/sub-test-model-7d2f73-kserve-778587d545-vqp6x" event={"ID":"779d3319-bb85-4d62-a1eb-3c27d4339c88","Type":"ContainerDied","Data":"fd45f661e87b2ccb6d2e4dc133365a57cd1e720e58455b28f7289db23aa9c324"} Jul 16 22:42:37.438603 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:37.438273 2585 scope.go:117] "RemoveContainer" containerID="ccb93f8931086bee5bf90d5f5b5f2d069ed9bd42cc43c1c2e3f0bb881656986a" Jul 16 22:42:37.447918 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:37.447901 2585 scope.go:117] "RemoveContainer" containerID="ccb93f8931086bee5bf90d5f5b5f2d069ed9bd42cc43c1c2e3f0bb881656986a" Jul 16 22:42:37.448175 ip-10-0-128-255 kubenswrapper[2585]: E0716 22:42:37.448155 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccb93f8931086bee5bf90d5f5b5f2d069ed9bd42cc43c1c2e3f0bb881656986a\": container with ID starting with ccb93f8931086bee5bf90d5f5b5f2d069ed9bd42cc43c1c2e3f0bb881656986a not found: ID does not exist" containerID="ccb93f8931086bee5bf90d5f5b5f2d069ed9bd42cc43c1c2e3f0bb881656986a" Jul 16 22:42:37.448244 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:37.448183 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccb93f8931086bee5bf90d5f5b5f2d069ed9bd42cc43c1c2e3f0bb881656986a"} err="failed to get container status \"ccb93f8931086bee5bf90d5f5b5f2d069ed9bd42cc43c1c2e3f0bb881656986a\": rpc error: code = NotFound desc = could not find container \"ccb93f8931086bee5bf90d5f5b5f2d069ed9bd42cc43c1c2e3f0bb881656986a\": container with ID starting with ccb93f8931086bee5bf90d5f5b5f2d069ed9bd42cc43c1c2e3f0bb881656986a not found: ID does not exist" Jul 16 22:42:37.450725 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:37.450708 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-1c9596/rate-test-model-1c9596-kserve-8c46746c9-9h9tm" Jul 16 22:42:37.460507 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:37.460488 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-7d2f73/sub-test-model-7d2f73-kserve-778587d545-vqp6x"] Jul 16 22:42:37.467187 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:37.467163 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-7d2f73/sub-test-model-7d2f73-kserve-778587d545-vqp6x"] Jul 16 22:42:37.934956 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:37.934925 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="779d3319-bb85-4d62-a1eb-3c27d4339c88" path="/var/lib/kubelet/pods/779d3319-bb85-4d62-a1eb-3c27d4339c88/volumes" Jul 16 22:42:49.273310 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:49.273279 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:42:49.376343 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:49.376313 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:42:49.789446 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:49.789416 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:42:54.803673 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:42:54.803578 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:43:10.417382 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:10.417344 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:43:20.430298 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:20.430201 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:43:21.790052 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:21.790023 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:43:26.214656 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:26.214618 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:43:29.366590 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:29.366557 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:43:29.468741 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:29.468712 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:43:34.244341 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:34.244305 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1c9596/rate-test-model-1c9596-kserve-8c46746c9-9h9tm"] Jul 16 22:43:34.268665 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:34.268635 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:43:35.119431 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:35.119381 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:43:36.020331 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:36.020290 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7472-139852394007216"] Jul 16 22:43:36.020853 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:36.020832 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="779d3319-bb85-4d62-a1eb-3c27d4339c88" containerName="main" Jul 16 22:43:36.020938 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:36.020855 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="779d3319-bb85-4d62-a1eb-3c27d4339c88" containerName="main" Jul 16 22:43:36.020996 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:36.020964 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="779d3319-bb85-4d62-a1eb-3c27d4339c88" containerName="main" Jul 16 22:43:36.024890 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:36.024867 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7472-139852394007216" Jul 16 22:43:36.027959 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:36.027939 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-b2kkw\"" Jul 16 22:43:36.032274 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:36.031928 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7472-139852394007216"] Jul 16 22:43:36.162715 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:36.162665 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcx5d\" (UniqueName: \"kubernetes.io/projected/bbfbeca8-e552-4ef3-9ebc-d06aab254a08-kube-api-access-tcx5d\") pod \"test-curl-7472-139852394007216\" (UID: \"bbfbeca8-e552-4ef3-9ebc-d06aab254a08\") " pod="opendatahub/test-curl-7472-139852394007216" Jul 16 22:43:36.264039 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:36.263997 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcx5d\" (UniqueName: \"kubernetes.io/projected/bbfbeca8-e552-4ef3-9ebc-d06aab254a08-kube-api-access-tcx5d\") pod \"test-curl-7472-139852394007216\" (UID: \"bbfbeca8-e552-4ef3-9ebc-d06aab254a08\") " pod="opendatahub/test-curl-7472-139852394007216" Jul 16 22:43:36.273955 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:36.273888 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcx5d\" (UniqueName: \"kubernetes.io/projected/bbfbeca8-e552-4ef3-9ebc-d06aab254a08-kube-api-access-tcx5d\") pod \"test-curl-7472-139852394007216\" (UID: \"bbfbeca8-e552-4ef3-9ebc-d06aab254a08\") " pod="opendatahub/test-curl-7472-139852394007216" Jul 16 22:43:36.334903 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:36.334878 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7472-139852394007216" Jul 16 22:43:36.475164 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:36.475138 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7472-139852394007216"] Jul 16 22:43:36.476856 ip-10-0-128-255 kubenswrapper[2585]: W0716 22:43:36.476832 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbbfbeca8_e552_4ef3_9ebc_d06aab254a08.slice/crio-44e8c216f344a4d0edc09080bc9e868cfc174ef11a3875fdb0b209fb149ace6f WatchSource:0}: Error finding container 44e8c216f344a4d0edc09080bc9e868cfc174ef11a3875fdb0b209fb149ace6f: Status 404 returned error can't find the container with id 44e8c216f344a4d0edc09080bc9e868cfc174ef11a3875fdb0b209fb149ace6f Jul 16 22:43:36.478163 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:36.478145 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 22:43:36.751132 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:36.751098 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7472-139852394007216" event={"ID":"bbfbeca8-e552-4ef3-9ebc-d06aab254a08","Type":"ContainerStarted","Data":"44e8c216f344a4d0edc09080bc9e868cfc174ef11a3875fdb0b209fb149ace6f"} Jul 16 22:43:37.758680 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:37.758650 2585 generic.go:356] "Generic (PLEG): container finished" podID="bbfbeca8-e552-4ef3-9ebc-d06aab254a08" containerID="2fc2b3bee387e35c2445a02309867db23be1b3a374a3fecfab25347e518a603b" exitCode=0 Jul 16 22:43:37.759067 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:37.758735 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7472-139852394007216" event={"ID":"bbfbeca8-e552-4ef3-9ebc-d06aab254a08","Type":"ContainerDied","Data":"2fc2b3bee387e35c2445a02309867db23be1b3a374a3fecfab25347e518a603b"} Jul 16 22:43:38.891176 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:38.891155 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7472-139852394007216" Jul 16 22:43:38.925898 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:38.925875 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7472-139852394007216_bbfbeca8-e552-4ef3-9ebc-d06aab254a08/test-curl-7472-139852394007216/0.log" Jul 16 22:43:38.963796 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:38.963771 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7472-139852394007216"] Jul 16 22:43:38.968103 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:38.968077 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7472-139852394007216"] Jul 16 22:43:38.987119 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:38.987098 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tcx5d\" (UniqueName: \"kubernetes.io/projected/bbfbeca8-e552-4ef3-9ebc-d06aab254a08-kube-api-access-tcx5d\") pod \"bbfbeca8-e552-4ef3-9ebc-d06aab254a08\" (UID: \"bbfbeca8-e552-4ef3-9ebc-d06aab254a08\") " Jul 16 22:43:38.989227 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:38.989198 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbfbeca8-e552-4ef3-9ebc-d06aab254a08-kube-api-access-tcx5d" (OuterVolumeSpecName: "kube-api-access-tcx5d") pod "bbfbeca8-e552-4ef3-9ebc-d06aab254a08" (UID: "bbfbeca8-e552-4ef3-9ebc-d06aab254a08"). InnerVolumeSpecName "kube-api-access-tcx5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 22:43:39.087887 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:39.087862 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcx5d\" (UniqueName: \"kubernetes.io/projected/bbfbeca8-e552-4ef3-9ebc-d06aab254a08-kube-api-access-tcx5d\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:43:39.770338 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:39.770310 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44e8c216f344a4d0edc09080bc9e868cfc174ef11a3875fdb0b209fb149ace6f" Jul 16 22:43:39.770338 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:39.770331 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7472-139852394007216" Jul 16 22:43:39.779840 ip-10-0-128-255 kubenswrapper[2585]: E0716 22:43:39.779812 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7472-139852394007216\" is forbidden: User \"system:node:ip-10-0-128-255.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-255.ec2.internal' and this object" podUID="bbfbeca8-e552-4ef3-9ebc-d06aab254a08" pod="opendatahub/test-curl-7472-139852394007216" Jul 16 22:43:39.933873 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:39.933846 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbfbeca8-e552-4ef3-9ebc-d06aab254a08" path="/var/lib/kubelet/pods/bbfbeca8-e552-4ef3-9ebc-d06aab254a08/volumes" Jul 16 22:43:40.526485 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:40.526448 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:43:49.264060 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:49.264013 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-1c9596/rate-test-model-1c9596-kserve-8c46746c9-9h9tm" podUID="e87ee267-229e-49f6-a835-ea50db8cb5ea" containerName="main" containerID="cri-o://1ab362bb4707644073b3d87b3cb30dc24d9d4ece2970cb384e9547f3b588a5c2" gracePeriod=45 Jul 16 22:43:49.505052 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:49.505030 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1c9596/rate-test-model-1c9596-kserve-8c46746c9-9h9tm" Jul 16 22:43:49.683840 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:49.683798 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e87ee267-229e-49f6-a835-ea50db8cb5ea-home\") pod \"e87ee267-229e-49f6-a835-ea50db8cb5ea\" (UID: \"e87ee267-229e-49f6-a835-ea50db8cb5ea\") " Jul 16 22:43:49.683840 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:49.683839 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e87ee267-229e-49f6-a835-ea50db8cb5ea-tmp-dir\") pod \"e87ee267-229e-49f6-a835-ea50db8cb5ea\" (UID: \"e87ee267-229e-49f6-a835-ea50db8cb5ea\") " Jul 16 22:43:49.684106 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:49.683870 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e87ee267-229e-49f6-a835-ea50db8cb5ea-dshm\") pod \"e87ee267-229e-49f6-a835-ea50db8cb5ea\" (UID: \"e87ee267-229e-49f6-a835-ea50db8cb5ea\") " Jul 16 22:43:49.684106 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:49.683889 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e87ee267-229e-49f6-a835-ea50db8cb5ea-tls-certs\") pod \"e87ee267-229e-49f6-a835-ea50db8cb5ea\" (UID: \"e87ee267-229e-49f6-a835-ea50db8cb5ea\") " Jul 16 22:43:49.684106 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:49.683921 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gq8c\" (UniqueName: \"kubernetes.io/projected/e87ee267-229e-49f6-a835-ea50db8cb5ea-kube-api-access-6gq8c\") pod \"e87ee267-229e-49f6-a835-ea50db8cb5ea\" (UID: \"e87ee267-229e-49f6-a835-ea50db8cb5ea\") " Jul 16 22:43:49.684106 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:49.683960 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e87ee267-229e-49f6-a835-ea50db8cb5ea-model-cache\") pod \"e87ee267-229e-49f6-a835-ea50db8cb5ea\" (UID: \"e87ee267-229e-49f6-a835-ea50db8cb5ea\") " Jul 16 22:43:49.684342 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:49.684135 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e87ee267-229e-49f6-a835-ea50db8cb5ea-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e87ee267-229e-49f6-a835-ea50db8cb5ea" (UID: "e87ee267-229e-49f6-a835-ea50db8cb5ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:43:49.684342 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:49.684147 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e87ee267-229e-49f6-a835-ea50db8cb5ea-home" (OuterVolumeSpecName: "home") pod "e87ee267-229e-49f6-a835-ea50db8cb5ea" (UID: "e87ee267-229e-49f6-a835-ea50db8cb5ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:43:49.684342 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:49.684301 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e87ee267-229e-49f6-a835-ea50db8cb5ea-model-cache" (OuterVolumeSpecName: "model-cache") pod "e87ee267-229e-49f6-a835-ea50db8cb5ea" (UID: "e87ee267-229e-49f6-a835-ea50db8cb5ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:43:49.684471 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:49.684350 2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e87ee267-229e-49f6-a835-ea50db8cb5ea-home\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:43:49.684471 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:49.684365 2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e87ee267-229e-49f6-a835-ea50db8cb5ea-tmp-dir\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:43:49.686167 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:49.686147 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e87ee267-229e-49f6-a835-ea50db8cb5ea-dshm" (OuterVolumeSpecName: "dshm") pod "e87ee267-229e-49f6-a835-ea50db8cb5ea" (UID: "e87ee267-229e-49f6-a835-ea50db8cb5ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:43:49.686258 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:49.686206 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e87ee267-229e-49f6-a835-ea50db8cb5ea-kube-api-access-6gq8c" (OuterVolumeSpecName: "kube-api-access-6gq8c") pod "e87ee267-229e-49f6-a835-ea50db8cb5ea" (UID: "e87ee267-229e-49f6-a835-ea50db8cb5ea"). InnerVolumeSpecName "kube-api-access-6gq8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 22:43:49.686303 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:49.686288 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e87ee267-229e-49f6-a835-ea50db8cb5ea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e87ee267-229e-49f6-a835-ea50db8cb5ea" (UID: "e87ee267-229e-49f6-a835-ea50db8cb5ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 22:43:49.784930 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:49.784892 2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e87ee267-229e-49f6-a835-ea50db8cb5ea-dshm\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:43:49.784930 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:49.784927 2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e87ee267-229e-49f6-a835-ea50db8cb5ea-tls-certs\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:43:49.785078 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:49.784943 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gq8c\" (UniqueName: \"kubernetes.io/projected/e87ee267-229e-49f6-a835-ea50db8cb5ea-kube-api-access-6gq8c\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:43:49.785078 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:49.784956 2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e87ee267-229e-49f6-a835-ea50db8cb5ea-model-cache\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:43:49.822412 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:49.822380 2585 generic.go:356] "Generic (PLEG): container finished" podID="e87ee267-229e-49f6-a835-ea50db8cb5ea" containerID="1ab362bb4707644073b3d87b3cb30dc24d9d4ece2970cb384e9547f3b588a5c2" exitCode=0 Jul 16 22:43:49.822583 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:49.822427 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1c9596/rate-test-model-1c9596-kserve-8c46746c9-9h9tm" Jul 16 22:43:49.822583 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:49.822473 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1c9596/rate-test-model-1c9596-kserve-8c46746c9-9h9tm" event={"ID":"e87ee267-229e-49f6-a835-ea50db8cb5ea","Type":"ContainerDied","Data":"1ab362bb4707644073b3d87b3cb30dc24d9d4ece2970cb384e9547f3b588a5c2"} Jul 16 22:43:49.822583 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:49.822529 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1c9596/rate-test-model-1c9596-kserve-8c46746c9-9h9tm" event={"ID":"e87ee267-229e-49f6-a835-ea50db8cb5ea","Type":"ContainerDied","Data":"016ae36a26ce69ee2046c1a1ea873076aa2abc48eb6ef4a844cb877dd80e3827"} Jul 16 22:43:49.822583 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:49.822554 2585 scope.go:117] "RemoveContainer" containerID="1ab362bb4707644073b3d87b3cb30dc24d9d4ece2970cb384e9547f3b588a5c2" Jul 16 22:43:49.832318 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:49.832300 2585 scope.go:117] "RemoveContainer" containerID="1ab362bb4707644073b3d87b3cb30dc24d9d4ece2970cb384e9547f3b588a5c2" Jul 16 22:43:49.832554 ip-10-0-128-255 kubenswrapper[2585]: E0716 22:43:49.832536 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ab362bb4707644073b3d87b3cb30dc24d9d4ece2970cb384e9547f3b588a5c2\": container with ID starting with 1ab362bb4707644073b3d87b3cb30dc24d9d4ece2970cb384e9547f3b588a5c2 not found: ID does not exist" containerID="1ab362bb4707644073b3d87b3cb30dc24d9d4ece2970cb384e9547f3b588a5c2" Jul 16 22:43:49.832616 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:49.832564 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ab362bb4707644073b3d87b3cb30dc24d9d4ece2970cb384e9547f3b588a5c2"} err="failed to get container status \"1ab362bb4707644073b3d87b3cb30dc24d9d4ece2970cb384e9547f3b588a5c2\": rpc error: code = NotFound desc = could not find container \"1ab362bb4707644073b3d87b3cb30dc24d9d4ece2970cb384e9547f3b588a5c2\": container with ID starting with 1ab362bb4707644073b3d87b3cb30dc24d9d4ece2970cb384e9547f3b588a5c2 not found: ID does not exist" Jul 16 22:43:49.845587 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:49.845559 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1c9596/rate-test-model-1c9596-kserve-8c46746c9-9h9tm"] Jul 16 22:43:49.850335 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:49.850313 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-1c9596/rate-test-model-1c9596-kserve-8c46746c9-9h9tm"] Jul 16 22:43:49.934014 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:49.933943 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e87ee267-229e-49f6-a835-ea50db8cb5ea" path="/var/lib/kubelet/pods/e87ee267-229e-49f6-a835-ea50db8cb5ea/volumes" Jul 16 22:43:54.824022 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:54.823980 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:43:54.920065 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:43:54.920026 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:44:05.037966 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:05.037919 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:44:08.246751 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:08.246719 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:44:09.399254 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:09.399207 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:44:17.127422 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:17.127367 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:44:17.363823 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:17.363785 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:44:20.248879 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:20.248846 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk"] Jul 16 22:44:20.249335 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:20.249027 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" podUID="8b28f8dc-5933-45b9-ad31-94a3492ee25e" containerName="istio-proxy" containerID="cri-o://c467b66e791e1d750feb29ef5ec010327812cfae683d08c5c298b0e40feb1f6b" gracePeriod=30 Jul 16 22:44:20.312505 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:20.312465 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:44:20.524604 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:20.524532 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:44:21.469797 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:21.469767 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:44:21.568323 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:21.568291 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:44:25.501419 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:25.501396 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" Jul 16 22:44:25.612962 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:25.612933 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jx8jb\" (UniqueName: \"kubernetes.io/projected/8b28f8dc-5933-45b9-ad31-94a3492ee25e-kube-api-access-jx8jb\") pod \"8b28f8dc-5933-45b9-ad31-94a3492ee25e\" (UID: \"8b28f8dc-5933-45b9-ad31-94a3492ee25e\") " Jul 16 22:44:25.613128 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:25.612970 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b28f8dc-5933-45b9-ad31-94a3492ee25e-credential-socket\") pod \"8b28f8dc-5933-45b9-ad31-94a3492ee25e\" (UID: \"8b28f8dc-5933-45b9-ad31-94a3492ee25e\") " Jul 16 22:44:25.613128 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:25.612996 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b28f8dc-5933-45b9-ad31-94a3492ee25e-istio-envoy\") pod \"8b28f8dc-5933-45b9-ad31-94a3492ee25e\" (UID: \"8b28f8dc-5933-45b9-ad31-94a3492ee25e\") " Jul 16 22:44:25.613128 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:25.613034 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b28f8dc-5933-45b9-ad31-94a3492ee25e-istio-podinfo\") pod \"8b28f8dc-5933-45b9-ad31-94a3492ee25e\" (UID: \"8b28f8dc-5933-45b9-ad31-94a3492ee25e\") " Jul 16 22:44:25.613128 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:25.613066 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b28f8dc-5933-45b9-ad31-94a3492ee25e-workload-certs\") pod \"8b28f8dc-5933-45b9-ad31-94a3492ee25e\" (UID: \"8b28f8dc-5933-45b9-ad31-94a3492ee25e\") " Jul 16 22:44:25.613128 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:25.613107 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b28f8dc-5933-45b9-ad31-94a3492ee25e-istio-token\") pod \"8b28f8dc-5933-45b9-ad31-94a3492ee25e\" (UID: \"8b28f8dc-5933-45b9-ad31-94a3492ee25e\") " Jul 16 22:44:25.613405 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:25.613137 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b28f8dc-5933-45b9-ad31-94a3492ee25e-istiod-ca-cert\") pod \"8b28f8dc-5933-45b9-ad31-94a3492ee25e\" (UID: \"8b28f8dc-5933-45b9-ad31-94a3492ee25e\") " Jul 16 22:44:25.613405 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:25.613181 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b28f8dc-5933-45b9-ad31-94a3492ee25e-istio-data\") pod \"8b28f8dc-5933-45b9-ad31-94a3492ee25e\" (UID: \"8b28f8dc-5933-45b9-ad31-94a3492ee25e\") " Jul 16 22:44:25.613405 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:25.613208 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b28f8dc-5933-45b9-ad31-94a3492ee25e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8b28f8dc-5933-45b9-ad31-94a3492ee25e" (UID: "8b28f8dc-5933-45b9-ad31-94a3492ee25e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:44:25.613552 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:25.613415 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b28f8dc-5933-45b9-ad31-94a3492ee25e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8b28f8dc-5933-45b9-ad31-94a3492ee25e" (UID: "8b28f8dc-5933-45b9-ad31-94a3492ee25e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:44:25.613679 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:25.613659 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b28f8dc-5933-45b9-ad31-94a3492ee25e-workload-socket\") pod \"8b28f8dc-5933-45b9-ad31-94a3492ee25e\" (UID: \"8b28f8dc-5933-45b9-ad31-94a3492ee25e\") " Jul 16 22:44:25.613941 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:25.613925 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b28f8dc-5933-45b9-ad31-94a3492ee25e-workload-certs\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:44:25.614006 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:25.613938 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b28f8dc-5933-45b9-ad31-94a3492ee25e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8b28f8dc-5933-45b9-ad31-94a3492ee25e" (UID: "8b28f8dc-5933-45b9-ad31-94a3492ee25e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:44:25.614006 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:25.613947 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b28f8dc-5933-45b9-ad31-94a3492ee25e-credential-socket\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:44:25.614160 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:25.614141 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b28f8dc-5933-45b9-ad31-94a3492ee25e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8b28f8dc-5933-45b9-ad31-94a3492ee25e" (UID: "8b28f8dc-5933-45b9-ad31-94a3492ee25e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 22:44:25.614271 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:25.614243 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b28f8dc-5933-45b9-ad31-94a3492ee25e-istio-data" (OuterVolumeSpecName: "istio-data") pod "8b28f8dc-5933-45b9-ad31-94a3492ee25e" (UID: "8b28f8dc-5933-45b9-ad31-94a3492ee25e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:44:25.615443 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:25.615416 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b28f8dc-5933-45b9-ad31-94a3492ee25e-istio-token" (OuterVolumeSpecName: "istio-token") pod "8b28f8dc-5933-45b9-ad31-94a3492ee25e" (UID: "8b28f8dc-5933-45b9-ad31-94a3492ee25e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 22:44:25.615443 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:25.615433 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b28f8dc-5933-45b9-ad31-94a3492ee25e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8b28f8dc-5933-45b9-ad31-94a3492ee25e" (UID: "8b28f8dc-5933-45b9-ad31-94a3492ee25e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:44:25.615707 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:25.615683 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b28f8dc-5933-45b9-ad31-94a3492ee25e-kube-api-access-jx8jb" (OuterVolumeSpecName: "kube-api-access-jx8jb") pod "8b28f8dc-5933-45b9-ad31-94a3492ee25e" (UID: "8b28f8dc-5933-45b9-ad31-94a3492ee25e"). InnerVolumeSpecName "kube-api-access-jx8jb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 22:44:25.615707 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:25.615683 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b28f8dc-5933-45b9-ad31-94a3492ee25e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8b28f8dc-5933-45b9-ad31-94a3492ee25e" (UID: "8b28f8dc-5933-45b9-ad31-94a3492ee25e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 22:44:25.715061 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:25.715026 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b28f8dc-5933-45b9-ad31-94a3492ee25e-workload-socket\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:44:25.715061 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:25.715056 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx8jb\" (UniqueName: \"kubernetes.io/projected/8b28f8dc-5933-45b9-ad31-94a3492ee25e-kube-api-access-jx8jb\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:44:25.715061 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:25.715066 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b28f8dc-5933-45b9-ad31-94a3492ee25e-istio-envoy\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:44:25.715312 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:25.715075 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b28f8dc-5933-45b9-ad31-94a3492ee25e-istio-podinfo\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:44:25.715312 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:25.715084 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b28f8dc-5933-45b9-ad31-94a3492ee25e-istio-token\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:44:25.715312 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:25.715091 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b28f8dc-5933-45b9-ad31-94a3492ee25e-istiod-ca-cert\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:44:25.715312 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:25.715100 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b28f8dc-5933-45b9-ad31-94a3492ee25e-istio-data\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:44:26.014724 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:26.014688 2585 generic.go:356] "Generic (PLEG): container finished" podID="8b28f8dc-5933-45b9-ad31-94a3492ee25e" containerID="c467b66e791e1d750feb29ef5ec010327812cfae683d08c5c298b0e40feb1f6b" exitCode=0 Jul 16 22:44:26.014927 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:26.014739 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" Jul 16 22:44:26.014927 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:26.014750 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" event={"ID":"8b28f8dc-5933-45b9-ad31-94a3492ee25e","Type":"ContainerDied","Data":"c467b66e791e1d750feb29ef5ec010327812cfae683d08c5c298b0e40feb1f6b"} Jul 16 22:44:26.014927 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:26.014794 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk" event={"ID":"8b28f8dc-5933-45b9-ad31-94a3492ee25e","Type":"ContainerDied","Data":"b67f04e998ef6176a001d85c773d553fc0da48530e044fc43c97ae85ac12e239"} Jul 16 22:44:26.014927 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:26.014814 2585 scope.go:117] "RemoveContainer" containerID="c467b66e791e1d750feb29ef5ec010327812cfae683d08c5c298b0e40feb1f6b" Jul 16 22:44:26.024238 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:26.024203 2585 scope.go:117] "RemoveContainer" containerID="3ca92af51e7cc51369554cd6f6f7fb0e4d810d25e201b3ad03377b84b4804597" Jul 16 22:44:26.032245 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:26.032206 2585 scope.go:117] "RemoveContainer" containerID="c467b66e791e1d750feb29ef5ec010327812cfae683d08c5c298b0e40feb1f6b" Jul 16 22:44:26.032507 ip-10-0-128-255 kubenswrapper[2585]: E0716 22:44:26.032485 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c467b66e791e1d750feb29ef5ec010327812cfae683d08c5c298b0e40feb1f6b\": container with ID starting with c467b66e791e1d750feb29ef5ec010327812cfae683d08c5c298b0e40feb1f6b not found: ID does not exist" containerID="c467b66e791e1d750feb29ef5ec010327812cfae683d08c5c298b0e40feb1f6b" Jul 16 22:44:26.032606 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:26.032517 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c467b66e791e1d750feb29ef5ec010327812cfae683d08c5c298b0e40feb1f6b"} err="failed to get container status \"c467b66e791e1d750feb29ef5ec010327812cfae683d08c5c298b0e40feb1f6b\": rpc error: code = NotFound desc = could not find container \"c467b66e791e1d750feb29ef5ec010327812cfae683d08c5c298b0e40feb1f6b\": container with ID starting with c467b66e791e1d750feb29ef5ec010327812cfae683d08c5c298b0e40feb1f6b not found: ID does not exist" Jul 16 22:44:26.032606 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:26.032539 2585 scope.go:117] "RemoveContainer" containerID="3ca92af51e7cc51369554cd6f6f7fb0e4d810d25e201b3ad03377b84b4804597" Jul 16 22:44:26.032828 ip-10-0-128-255 kubenswrapper[2585]: E0716 22:44:26.032806 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ca92af51e7cc51369554cd6f6f7fb0e4d810d25e201b3ad03377b84b4804597\": container with ID starting with 3ca92af51e7cc51369554cd6f6f7fb0e4d810d25e201b3ad03377b84b4804597 not found: ID does not exist" containerID="3ca92af51e7cc51369554cd6f6f7fb0e4d810d25e201b3ad03377b84b4804597" Jul 16 22:44:26.032927 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:26.032843 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ca92af51e7cc51369554cd6f6f7fb0e4d810d25e201b3ad03377b84b4804597"} err="failed to get container status \"3ca92af51e7cc51369554cd6f6f7fb0e4d810d25e201b3ad03377b84b4804597\": rpc error: code = NotFound desc = could not find container \"3ca92af51e7cc51369554cd6f6f7fb0e4d810d25e201b3ad03377b84b4804597\": container with ID starting with 3ca92af51e7cc51369554cd6f6f7fb0e4d810d25e201b3ad03377b84b4804597 not found: ID does not exist" Jul 16 22:44:26.036053 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:26.036032 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk"] Jul 16 22:44:26.043461 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:26.043440 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-1c9596-openshift-default-dd87855dc-7rqxk"] Jul 16 22:44:27.934966 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:27.934935 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b28f8dc-5933-45b9-ad31-94a3492ee25e" path="/var/lib/kubelet/pods/8b28f8dc-5933-45b9-ad31-94a3492ee25e/volumes" Jul 16 22:44:31.237571 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:31.237541 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:44:34.511167 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:34.511131 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cth87"] Jul 16 22:44:35.732653 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:35.732620 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:44:42.727819 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:42.727780 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:44:42.879764 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:42.879735 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:44:46.433200 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:46.433166 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g"] Jul 16 22:44:46.433693 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:46.433451 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" podUID="bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f" containerName="istio-proxy" containerID="cri-o://03b662f41a93f5358d3d2c1dc898211a39946bf198b174fadc60f6644e819727" gracePeriod=30 Jul 16 22:44:46.476596 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:46.476564 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:44:46.708933 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:46.708847 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:44:47.091597 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:47.091565 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-84df8b88d9-r55wz_3480c87a-2733-42df-83f1-8eaaa7287e84/authorino/0.log" Jul 16 22:44:51.685297 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:51.685274 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" Jul 16 22:44:51.727427 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:51.727360 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-istio-token\") pod \"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f\" (UID: \"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f\") " Jul 16 22:44:51.727427 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:51.727413 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-istiod-ca-cert\") pod \"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f\" (UID: \"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f\") " Jul 16 22:44:51.727607 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:51.727436 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-workload-socket\") pod \"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f\" (UID: \"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f\") " Jul 16 22:44:51.727607 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:51.727458 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-workload-certs\") pod \"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f\" (UID: \"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f\") " Jul 16 22:44:51.727607 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:51.727492 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-istio-data\") pod \"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f\" (UID: \"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f\") " Jul 16 22:44:51.727607 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:51.727563 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-istio-podinfo\") pod \"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f\" (UID: \"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f\") " Jul 16 22:44:51.727815 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:51.727613 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62dxf\" (UniqueName: \"kubernetes.io/projected/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-kube-api-access-62dxf\") pod \"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f\" (UID: \"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f\") " Jul 16 22:44:51.727815 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:51.727636 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-istio-envoy\") pod \"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f\" (UID: \"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f\") " Jul 16 22:44:51.727815 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:51.727668 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-credential-socket\") pod \"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f\" (UID: \"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f\") " Jul 16 22:44:51.727815 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:51.727737 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f" (UID: "bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:44:51.727815 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:51.727806 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f" (UID: "bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 22:44:51.728068 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:51.727824 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f" (UID: "bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:44:51.728068 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:51.727976 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f" (UID: "bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:44:51.728068 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:51.728025 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-istiod-ca-cert\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:44:51.728068 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:51.728047 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-workload-socket\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:44:51.728068 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:51.728063 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-workload-certs\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:44:51.728662 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:51.728629 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-istio-data" (OuterVolumeSpecName: "istio-data") pod "bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f" (UID: "bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:44:51.730045 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:51.730014 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f" (UID: "bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 22:44:51.730045 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:51.730038 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-istio-token" (OuterVolumeSpecName: "istio-token") pod "bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f" (UID: "bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 22:44:51.730310 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:51.730281 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f" (UID: "bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:44:51.730310 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:51.730288 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-kube-api-access-62dxf" (OuterVolumeSpecName: "kube-api-access-62dxf") pod "bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f" (UID: "bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f"). InnerVolumeSpecName "kube-api-access-62dxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 22:44:51.829158 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:51.829127 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-istio-data\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:44:51.829158 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:51.829153 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-istio-podinfo\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:44:51.829158 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:51.829163 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62dxf\" (UniqueName: \"kubernetes.io/projected/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-kube-api-access-62dxf\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:44:51.829443 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:51.829171 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-istio-envoy\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:44:51.829443 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:51.829180 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-credential-socket\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:44:51.829443 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:51.829187 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f-istio-token\") on node \"ip-10-0-128-255.ec2.internal\" DevicePath \"\"" Jul 16 22:44:52.155857 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:52.155827 2585 generic.go:356] "Generic (PLEG): container finished" podID="bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f" containerID="03b662f41a93f5358d3d2c1dc898211a39946bf198b174fadc60f6644e819727" exitCode=0 Jul 16 22:44:52.156006 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:52.155878 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" event={"ID":"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f","Type":"ContainerDied","Data":"03b662f41a93f5358d3d2c1dc898211a39946bf198b174fadc60f6644e819727"} Jul 16 22:44:52.156006 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:52.155882 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" Jul 16 22:44:52.156006 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:52.155920 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g" event={"ID":"bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f","Type":"ContainerDied","Data":"94a2729a767ffbb0594135df7a33e83d18af5827d69757eb62977e260b69f073"} Jul 16 22:44:52.156006 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:52.155942 2585 scope.go:117] "RemoveContainer" containerID="03b662f41a93f5358d3d2c1dc898211a39946bf198b174fadc60f6644e819727" Jul 16 22:44:52.165086 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:52.165069 2585 scope.go:117] "RemoveContainer" containerID="795b1754ac421823b8c878b7be7d63e8b7db7125a9af3f173c9f7c76bc21bed9" Jul 16 22:44:52.173651 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:52.173630 2585 scope.go:117] "RemoveContainer" containerID="03b662f41a93f5358d3d2c1dc898211a39946bf198b174fadc60f6644e819727" Jul 16 22:44:52.173946 ip-10-0-128-255 kubenswrapper[2585]: E0716 22:44:52.173926 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03b662f41a93f5358d3d2c1dc898211a39946bf198b174fadc60f6644e819727\": container with ID starting with 03b662f41a93f5358d3d2c1dc898211a39946bf198b174fadc60f6644e819727 not found: ID does not exist" containerID="03b662f41a93f5358d3d2c1dc898211a39946bf198b174fadc60f6644e819727" Jul 16 22:44:52.174035 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:52.173957 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03b662f41a93f5358d3d2c1dc898211a39946bf198b174fadc60f6644e819727"} err="failed to get container status \"03b662f41a93f5358d3d2c1dc898211a39946bf198b174fadc60f6644e819727\": rpc error: code = NotFound desc = could not find container \"03b662f41a93f5358d3d2c1dc898211a39946bf198b174fadc60f6644e819727\": container with ID starting with 03b662f41a93f5358d3d2c1dc898211a39946bf198b174fadc60f6644e819727 not found: ID does not exist" Jul 16 22:44:52.174035 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:52.173975 2585 scope.go:117] "RemoveContainer" containerID="795b1754ac421823b8c878b7be7d63e8b7db7125a9af3f173c9f7c76bc21bed9" Jul 16 22:44:52.174212 ip-10-0-128-255 kubenswrapper[2585]: E0716 22:44:52.174197 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"795b1754ac421823b8c878b7be7d63e8b7db7125a9af3f173c9f7c76bc21bed9\": container with ID starting with 795b1754ac421823b8c878b7be7d63e8b7db7125a9af3f173c9f7c76bc21bed9 not found: ID does not exist" containerID="795b1754ac421823b8c878b7be7d63e8b7db7125a9af3f173c9f7c76bc21bed9" Jul 16 22:44:52.174212 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:52.174230 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"795b1754ac421823b8c878b7be7d63e8b7db7125a9af3f173c9f7c76bc21bed9"} err="failed to get container status \"795b1754ac421823b8c878b7be7d63e8b7db7125a9af3f173c9f7c76bc21bed9\": rpc error: code = NotFound desc = could not find container \"795b1754ac421823b8c878b7be7d63e8b7db7125a9af3f173c9f7c76bc21bed9\": container with ID starting with 795b1754ac421823b8c878b7be7d63e8b7db7125a9af3f173c9f7c76bc21bed9 not found: ID does not exist" Jul 16 22:44:52.174613 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:52.174594 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g"] Jul 16 22:44:52.180908 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:52.180888 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-7d2f73-openshift-default-695c464cb7-zgx4g"] Jul 16 22:44:53.648805 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:53.648773 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-xg9kf_40d4f474-fbc3-4196-bf87-8a3510483609/server/0.log" Jul 16 22:44:53.935039 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:53.934954 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f" path="/var/lib/kubelet/pods/bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f/volumes" Jul 16 22:44:54.169398 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:54.169370 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-574666dc8f-7dmxp_f84b0d43-3571-419d-8232-2f7418f2da3a/copy-manifests/0.log" Jul 16 22:44:54.177849 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:54.177817 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-574666dc8f-7dmxp_f84b0d43-3571-419d-8232-2f7418f2da3a/manager/0.log" Jul 16 22:44:56.863192 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:56.863161 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596njrj_8bbdf3c4-c287-4af3-be2c-6af952e8a9c2/pull/0.log" Jul 16 22:44:56.870009 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:56.869987 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596njrj_8bbdf3c4-c287-4af3-be2c-6af952e8a9c2/extract/0.log" Jul 16 22:44:56.877320 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:56.877298 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596njrj_8bbdf3c4-c287-4af3-be2c-6af952e8a9c2/util/0.log" Jul 16 22:44:56.984939 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:56.984909 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e089f4p_bc100a65-eb66-4a34-9e07-7fd60fe7d3f5/util/0.log" Jul 16 22:44:56.994810 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:56.994789 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e089f4p_bc100a65-eb66-4a34-9e07-7fd60fe7d3f5/pull/0.log" Jul 16 22:44:57.001555 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:57.001535 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e089f4p_bc100a65-eb66-4a34-9e07-7fd60fe7d3f5/extract/0.log" Jul 16 22:44:57.104972 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:57.104944 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739qtkh_7b6c3b0d-c66b-46ee-b013-b5bb9e619888/util/0.log" Jul 16 22:44:57.111551 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:57.111527 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739qtkh_7b6c3b0d-c66b-46ee-b013-b5bb9e619888/pull/0.log" Jul 16 22:44:57.117957 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:57.117880 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739qtkh_7b6c3b0d-c66b-46ee-b013-b5bb9e619888/extract/0.log" Jul 16 22:44:57.228925 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:57.228890 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fvbjb_1deb487d-8753-4dc6-a838-2c20e9841647/pull/0.log" Jul 16 22:44:57.239938 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:57.239911 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fvbjb_1deb487d-8753-4dc6-a838-2c20e9841647/extract/0.log" Jul 16 22:44:57.247672 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:57.247649 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fvbjb_1deb487d-8753-4dc6-a838-2c20e9841647/util/0.log" Jul 16 22:44:57.366513 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:57.366480 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-84df8b88d9-r55wz_3480c87a-2733-42df-83f1-8eaaa7287e84/authorino/0.log" Jul 16 22:44:58.058706 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:58.058671 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-cth87_33460968-f59d-4d7f-af09-731b58eaf8de/limitador/0.log" Jul 16 22:44:58.945053 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:44:58.945025 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54f57477db-cw8lx_2db26749-c5a9-4647-87f0-de77f6ff2f3f/kube-auth-proxy/0.log" Jul 16 22:45:00.525158 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:00.525134 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-5dvvb_b7053ba1-14fe-4849-ad04-509946c1e37b/main/0.log" Jul 16 22:45:05.624732 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:05.624685 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xsq47/must-gather-c9fgf"] Jul 16 22:45:05.625123 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:05.625039 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b28f8dc-5933-45b9-ad31-94a3492ee25e" containerName="istio-proxy" Jul 16 22:45:05.625123 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:05.625050 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b28f8dc-5933-45b9-ad31-94a3492ee25e" containerName="istio-proxy" Jul 16 22:45:05.625123 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:05.625060 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e87ee267-229e-49f6-a835-ea50db8cb5ea" containerName="main" Jul 16 22:45:05.625123 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:05.625065 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e87ee267-229e-49f6-a835-ea50db8cb5ea" containerName="main" Jul 16 22:45:05.625123 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:05.625077 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbfbeca8-e552-4ef3-9ebc-d06aab254a08" containerName="test-curl-7472-139852394007216" Jul 16 22:45:05.625123 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:05.625083 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbfbeca8-e552-4ef3-9ebc-d06aab254a08" containerName="test-curl-7472-139852394007216" Jul 16 22:45:05.625123 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:05.625092 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f" containerName="istio-proxy" Jul 16 22:45:05.625123 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:05.625098 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f" containerName="istio-proxy" Jul 16 22:45:05.625123 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:05.625109 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f" containerName="istio-proxy" Jul 16 22:45:05.625123 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:05.625114 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f" containerName="istio-proxy" Jul 16 22:45:05.625466 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:05.625168 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f" containerName="istio-proxy" Jul 16 22:45:05.625466 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:05.625175 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e87ee267-229e-49f6-a835-ea50db8cb5ea" containerName="main" Jul 16 22:45:05.625466 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:05.625183 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="bba4d0a2-ac2b-47f5-b7b5-d7d9a8203c6f" containerName="istio-proxy" Jul 16 22:45:05.625466 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:05.625189 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b28f8dc-5933-45b9-ad31-94a3492ee25e" containerName="istio-proxy" Jul 16 22:45:05.625466 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:05.625195 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b28f8dc-5933-45b9-ad31-94a3492ee25e" containerName="istio-proxy" Jul 16 22:45:05.625466 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:05.625200 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbfbeca8-e552-4ef3-9ebc-d06aab254a08" containerName="test-curl-7472-139852394007216" Jul 16 22:45:05.625466 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:05.625280 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b28f8dc-5933-45b9-ad31-94a3492ee25e" containerName="istio-proxy" Jul 16 22:45:05.625466 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:05.625287 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b28f8dc-5933-45b9-ad31-94a3492ee25e" containerName="istio-proxy" Jul 16 22:45:05.629853 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:05.629831 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xsq47/must-gather-c9fgf" Jul 16 22:45:05.632360 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:05.632329 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xsq47\"/\"default-dockercfg-8flnx\"" Jul 16 22:45:05.632360 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:05.632350 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xsq47\"/\"kube-root-ca.crt\"" Jul 16 22:45:05.632531 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:05.632338 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xsq47\"/\"openshift-service-ca.crt\"" Jul 16 22:45:05.638957 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:05.638932 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xsq47/must-gather-c9fgf"] Jul 16 22:45:05.641314 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:05.641287 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:45:05.738910 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:05.738878 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" limit=1024 Jul 16 22:45:05.748071 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:05.748050 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/64b7e0f0-b205-4ae4-8bbe-3b46d63788d4-must-gather-output\") pod \"must-gather-c9fgf\" (UID: \"64b7e0f0-b205-4ae4-8bbe-3b46d63788d4\") " pod="openshift-must-gather-xsq47/must-gather-c9fgf" Jul 16 22:45:05.748143 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:05.748118 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7sck\" (UniqueName: \"kubernetes.io/projected/64b7e0f0-b205-4ae4-8bbe-3b46d63788d4-kube-api-access-n7sck\") pod \"must-gather-c9fgf\" (UID: \"64b7e0f0-b205-4ae4-8bbe-3b46d63788d4\") " pod="openshift-must-gather-xsq47/must-gather-c9fgf" Jul 16 22:45:05.849288 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:05.849243 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7sck\" (UniqueName: \"kubernetes.io/projected/64b7e0f0-b205-4ae4-8bbe-3b46d63788d4-kube-api-access-n7sck\") pod \"must-gather-c9fgf\" (UID: \"64b7e0f0-b205-4ae4-8bbe-3b46d63788d4\") " pod="openshift-must-gather-xsq47/must-gather-c9fgf" Jul 16 22:45:05.849436 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:05.849316 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/64b7e0f0-b205-4ae4-8bbe-3b46d63788d4-must-gather-output\") pod \"must-gather-c9fgf\" (UID: \"64b7e0f0-b205-4ae4-8bbe-3b46d63788d4\") " pod="openshift-must-gather-xsq47/must-gather-c9fgf" Jul 16 22:45:05.849666 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:05.849644 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/64b7e0f0-b205-4ae4-8bbe-3b46d63788d4-must-gather-output\") pod \"must-gather-c9fgf\" (UID: \"64b7e0f0-b205-4ae4-8bbe-3b46d63788d4\") " pod="openshift-must-gather-xsq47/must-gather-c9fgf" Jul 16 22:45:05.865796 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:05.865768 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7sck\" (UniqueName: \"kubernetes.io/projected/64b7e0f0-b205-4ae4-8bbe-3b46d63788d4-kube-api-access-n7sck\") pod \"must-gather-c9fgf\" (UID: \"64b7e0f0-b205-4ae4-8bbe-3b46d63788d4\") " pod="openshift-must-gather-xsq47/must-gather-c9fgf" Jul 16 22:45:05.940060 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:05.939984 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xsq47/must-gather-c9fgf" Jul 16 22:45:06.092409 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:06.092347 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xsq47/must-gather-c9fgf"] Jul 16 22:45:06.095369 ip-10-0-128-255 kubenswrapper[2585]: W0716 22:45:06.095331 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64b7e0f0_b205_4ae4_8bbe_3b46d63788d4.slice/crio-d7818ce14cfddfc35201ed62001616d08d4ea987a037c954556524d267a0c042 WatchSource:0}: Error finding container d7818ce14cfddfc35201ed62001616d08d4ea987a037c954556524d267a0c042: Status 404 returned error can't find the container with id d7818ce14cfddfc35201ed62001616d08d4ea987a037c954556524d267a0c042 Jul 16 22:45:06.238834 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:06.238749 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xsq47/must-gather-c9fgf" event={"ID":"64b7e0f0-b205-4ae4-8bbe-3b46d63788d4","Type":"ContainerStarted","Data":"d7818ce14cfddfc35201ed62001616d08d4ea987a037c954556524d267a0c042"} Jul 16 22:45:07.248972 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:07.248874 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xsq47/must-gather-c9fgf" event={"ID":"64b7e0f0-b205-4ae4-8bbe-3b46d63788d4","Type":"ContainerStarted","Data":"e7ea1234e295f0a6b903e5559879594306def2df6d8c45a89271374fd5c28d2d"} Jul 16 22:45:07.248972 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:07.248930 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xsq47/must-gather-c9fgf" event={"ID":"64b7e0f0-b205-4ae4-8bbe-3b46d63788d4","Type":"ContainerStarted","Data":"ddfbdd55d445e2866e60d5719c326bfb651a5c6990588d998ecd31fc7a334f34"} Jul 16 22:45:07.268123 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:07.268050 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xsq47/must-gather-c9fgf" podStartSLOduration=1.458828473 podStartE2EDuration="2.268029702s" totalImagesPullingTime="809.201229ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:45:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:45:07.265533903 +0000 UTC m=+3455.887976856" watchObservedRunningTime="2026-07-16 22:45:07.268029702 +0000 UTC m=+3455.890472656" Jul 16 22:45:08.587575 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:08.587549 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jwlrc_e9b9235a-c4fc-4aac-93b4-2964356667f1/global-pull-secret-syncer/0.log" Jul 16 22:45:08.733632 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:08.733592 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lfwng_43f80a88-5e25-4f60-980a-8bb22936ba6c/konnectivity-agent/0.log" Jul 16 22:45:08.870743 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:08.870649 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-255.ec2.internal_9c641fdc20deb5fa481c19356c3eeb80/haproxy/0.log" Jul 16 22:45:13.578370 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:13.578331 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596njrj_8bbdf3c4-c287-4af3-be2c-6af952e8a9c2/extract/0.log" Jul 16 22:45:13.614454 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:13.614421 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596njrj_8bbdf3c4-c287-4af3-be2c-6af952e8a9c2/util/0.log" Jul 16 22:45:13.651392 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:13.651357 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596njrj_8bbdf3c4-c287-4af3-be2c-6af952e8a9c2/pull/0.log" Jul 16 22:45:13.689929 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:13.689891 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e089f4p_bc100a65-eb66-4a34-9e07-7fd60fe7d3f5/extract/0.log" Jul 16 22:45:13.728727 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:13.728693 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e089f4p_bc100a65-eb66-4a34-9e07-7fd60fe7d3f5/util/0.log" Jul 16 22:45:13.755665 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:13.755630 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e089f4p_bc100a65-eb66-4a34-9e07-7fd60fe7d3f5/pull/0.log" Jul 16 22:45:13.792123 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:13.792087 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739qtkh_7b6c3b0d-c66b-46ee-b013-b5bb9e619888/extract/0.log" Jul 16 22:45:13.834268 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:13.834163 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739qtkh_7b6c3b0d-c66b-46ee-b013-b5bb9e619888/util/0.log" Jul 16 22:45:13.856687 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:13.856652 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739qtkh_7b6c3b0d-c66b-46ee-b013-b5bb9e619888/pull/0.log" Jul 16 22:45:13.890779 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:13.890617 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fvbjb_1deb487d-8753-4dc6-a838-2c20e9841647/extract/0.log" Jul 16 22:45:13.922194 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:13.922114 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fvbjb_1deb487d-8753-4dc6-a838-2c20e9841647/util/0.log" Jul 16 22:45:13.952386 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:13.952348 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fvbjb_1deb487d-8753-4dc6-a838-2c20e9841647/pull/0.log" Jul 16 22:45:14.402386 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:14.402263 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-84df8b88d9-r55wz_3480c87a-2733-42df-83f1-8eaaa7287e84/authorino/0.log" Jul 16 22:45:14.759297 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:14.759156 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-cth87_33460968-f59d-4d7f-af09-731b58eaf8de/limitador/0.log" Jul 16 22:45:16.570533 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:16.570501 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9fcf5399-29b4-46a1-84ed-2779ac199ead/alertmanager/0.log" Jul 16 22:45:16.613318 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:16.613290 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9fcf5399-29b4-46a1-84ed-2779ac199ead/config-reloader/0.log" Jul 16 22:45:16.647674 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:16.647644 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9fcf5399-29b4-46a1-84ed-2779ac199ead/kube-rbac-proxy-web/0.log" Jul 16 22:45:16.695579 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:16.695554 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9fcf5399-29b4-46a1-84ed-2779ac199ead/kube-rbac-proxy/0.log" Jul 16 22:45:16.726653 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:16.726623 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9fcf5399-29b4-46a1-84ed-2779ac199ead/kube-rbac-proxy-metric/0.log" Jul 16 22:45:16.758054 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:16.758021 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9fcf5399-29b4-46a1-84ed-2779ac199ead/prom-label-proxy/0.log" Jul 16 22:45:16.790879 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:16.790840 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9fcf5399-29b4-46a1-84ed-2779ac199ead/init-config-reloader/0.log" Jul 16 22:45:16.934671 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:16.934585 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-gt52v_c82e3280-90dc-4a6f-858f-66024893e398/kube-state-metrics/0.log" Jul 16 22:45:16.974831 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:16.974800 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-gt52v_c82e3280-90dc-4a6f-858f-66024893e398/kube-rbac-proxy-main/0.log" Jul 16 22:45:17.023894 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:17.023865 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-gt52v_c82e3280-90dc-4a6f-858f-66024893e398/kube-rbac-proxy-self/0.log" Jul 16 22:45:17.095248 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:17.095207 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5d897c97d7-c8rk6_5485377e-20a2-4f1e-b335-8eed2032a4bf/metrics-server/0.log" Jul 16 22:45:17.195109 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:17.195025 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6ww9h_4ab63d4c-dc11-4ed8-8f34-4dfcce97f939/node-exporter/0.log" Jul 16 22:45:17.227989 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:17.227954 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6ww9h_4ab63d4c-dc11-4ed8-8f34-4dfcce97f939/kube-rbac-proxy/0.log" Jul 16 22:45:17.259061 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:17.259033 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6ww9h_4ab63d4c-dc11-4ed8-8f34-4dfcce97f939/init-textfile/0.log" Jul 16 22:45:17.858085 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:17.858055 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-zcmjn_bc97620a-757d-4bcb-ab6c-6a021f675d55/prometheus-operator-admission-webhook/0.log" Jul 16 22:45:17.891490 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:17.891464 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5ff499df55-jnr4s_a35bdd2d-04ea-4c54-84e9-bece735a80ff/telemeter-client/0.log" Jul 16 22:45:17.922892 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:17.922864 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5ff499df55-jnr4s_a35bdd2d-04ea-4c54-84e9-bece735a80ff/reload/0.log" Jul 16 22:45:17.953347 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:17.953313 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5ff499df55-jnr4s_a35bdd2d-04ea-4c54-84e9-bece735a80ff/kube-rbac-proxy/0.log" Jul 16 22:45:18.291179 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:18.291090 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xsq47/perf-node-gather-daemonset-mm8vb"] Jul 16 22:45:18.297704 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:18.297671 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-mm8vb" Jul 16 22:45:18.315874 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:18.315839 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xsq47/perf-node-gather-daemonset-mm8vb"] Jul 16 22:45:18.387679 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:18.387635 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f5659bb3-b2ee-4328-9337-4183932e5b66-sys\") pod \"perf-node-gather-daemonset-mm8vb\" (UID: \"f5659bb3-b2ee-4328-9337-4183932e5b66\") " pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-mm8vb" Jul 16 22:45:18.387889 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:18.387687 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f5659bb3-b2ee-4328-9337-4183932e5b66-podres\") pod \"perf-node-gather-daemonset-mm8vb\" (UID: \"f5659bb3-b2ee-4328-9337-4183932e5b66\") " pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-mm8vb" Jul 16 22:45:18.387889 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:18.387773 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f5659bb3-b2ee-4328-9337-4183932e5b66-lib-modules\") pod \"perf-node-gather-daemonset-mm8vb\" (UID: \"f5659bb3-b2ee-4328-9337-4183932e5b66\") " pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-mm8vb" Jul 16 22:45:18.387889 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:18.387822 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqwvv\" (UniqueName: \"kubernetes.io/projected/f5659bb3-b2ee-4328-9337-4183932e5b66-kube-api-access-gqwvv\") pod \"perf-node-gather-daemonset-mm8vb\" (UID: \"f5659bb3-b2ee-4328-9337-4183932e5b66\") " pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-mm8vb" Jul 16 22:45:18.387889 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:18.387870 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f5659bb3-b2ee-4328-9337-4183932e5b66-proc\") pod \"perf-node-gather-daemonset-mm8vb\" (UID: \"f5659bb3-b2ee-4328-9337-4183932e5b66\") " pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-mm8vb" Jul 16 22:45:18.489386 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:18.489333 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f5659bb3-b2ee-4328-9337-4183932e5b66-lib-modules\") pod \"perf-node-gather-daemonset-mm8vb\" (UID: \"f5659bb3-b2ee-4328-9337-4183932e5b66\") " pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-mm8vb" Jul 16 22:45:18.489594 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:18.489411 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqwvv\" (UniqueName: \"kubernetes.io/projected/f5659bb3-b2ee-4328-9337-4183932e5b66-kube-api-access-gqwvv\") pod \"perf-node-gather-daemonset-mm8vb\" (UID: \"f5659bb3-b2ee-4328-9337-4183932e5b66\") " pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-mm8vb" Jul 16 22:45:18.489594 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:18.489445 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f5659bb3-b2ee-4328-9337-4183932e5b66-proc\") pod \"perf-node-gather-daemonset-mm8vb\" (UID: \"f5659bb3-b2ee-4328-9337-4183932e5b66\") " pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-mm8vb" Jul 16 22:45:18.489594 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:18.489480 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f5659bb3-b2ee-4328-9337-4183932e5b66-sys\") pod \"perf-node-gather-daemonset-mm8vb\" (UID: \"f5659bb3-b2ee-4328-9337-4183932e5b66\") " pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-mm8vb" Jul 16 22:45:18.489594 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:18.489494 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f5659bb3-b2ee-4328-9337-4183932e5b66-lib-modules\") pod \"perf-node-gather-daemonset-mm8vb\" (UID: \"f5659bb3-b2ee-4328-9337-4183932e5b66\") " pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-mm8vb" Jul 16 22:45:18.489594 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:18.489550 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f5659bb3-b2ee-4328-9337-4183932e5b66-sys\") pod \"perf-node-gather-daemonset-mm8vb\" (UID: \"f5659bb3-b2ee-4328-9337-4183932e5b66\") " pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-mm8vb" Jul 16 22:45:18.489594 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:18.489550 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f5659bb3-b2ee-4328-9337-4183932e5b66-proc\") pod \"perf-node-gather-daemonset-mm8vb\" (UID: \"f5659bb3-b2ee-4328-9337-4183932e5b66\") " pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-mm8vb" Jul 16 22:45:18.489594 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:18.489503 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f5659bb3-b2ee-4328-9337-4183932e5b66-podres\") pod \"perf-node-gather-daemonset-mm8vb\" (UID: \"f5659bb3-b2ee-4328-9337-4183932e5b66\") " pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-mm8vb" Jul 16 22:45:18.489594 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:18.489577 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f5659bb3-b2ee-4328-9337-4183932e5b66-podres\") pod \"perf-node-gather-daemonset-mm8vb\" (UID: \"f5659bb3-b2ee-4328-9337-4183932e5b66\") " pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-mm8vb" Jul 16 22:45:18.498920 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:18.498884 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqwvv\" (UniqueName: \"kubernetes.io/projected/f5659bb3-b2ee-4328-9337-4183932e5b66-kube-api-access-gqwvv\") pod \"perf-node-gather-daemonset-mm8vb\" (UID: \"f5659bb3-b2ee-4328-9337-4183932e5b66\") " pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-mm8vb" Jul 16 22:45:18.611710 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:18.611674 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-mm8vb" Jul 16 22:45:18.779730 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:18.779698 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xsq47/perf-node-gather-daemonset-mm8vb"] Jul 16 22:45:18.780868 ip-10-0-128-255 kubenswrapper[2585]: W0716 22:45:18.780820 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf5659bb3_b2ee_4328_9337_4183932e5b66.slice/crio-34f4eecc82340eb363a36b8370702c9ed0dd33092e79c1d72d72fb85ec30518f WatchSource:0}: Error finding container 34f4eecc82340eb363a36b8370702c9ed0dd33092e79c1d72d72fb85ec30518f: Status 404 returned error can't find the container with id 34f4eecc82340eb363a36b8370702c9ed0dd33092e79c1d72d72fb85ec30518f Jul 16 22:45:19.343703 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:19.343662 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-mm8vb" event={"ID":"f5659bb3-b2ee-4328-9337-4183932e5b66","Type":"ContainerStarted","Data":"bba348bf889898708ffd1b5f2b3e5254f27de089bd0a76d9740427879e37421f"} Jul 16 22:45:19.344212 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:19.343739 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-mm8vb" event={"ID":"f5659bb3-b2ee-4328-9337-4183932e5b66","Type":"ContainerStarted","Data":"34f4eecc82340eb363a36b8370702c9ed0dd33092e79c1d72d72fb85ec30518f"} Jul 16 22:45:19.344212 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:19.343839 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-mm8vb" Jul 16 22:45:19.364060 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:19.363993 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-mm8vb" podStartSLOduration=1.363971237 podStartE2EDuration="1.363971237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:45:19.363211563 +0000 UTC m=+3467.985654517" watchObservedRunningTime="2026-07-16 22:45:19.363971237 +0000 UTC m=+3467.986414191" Jul 16 22:45:20.351079 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:20.351049 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-njjsn_68e6c5e2-0a59-422d-a1c8-9b735597ed43/download-server/0.log" Jul 16 22:45:22.170743 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:22.170702 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5hrrt_7b50691f-e781-4af6-bb30-6600349a399f/dns/0.log" Jul 16 22:45:22.257241 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:22.257166 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5hrrt_7b50691f-e781-4af6-bb30-6600349a399f/kube-rbac-proxy/0.log" Jul 16 22:45:22.431009 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:22.430940 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7tltv_1dabb80a-8529-4289-82e4-e44ca61494c8/dns-node-resolver/0.log" Jul 16 22:45:23.093702 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:23.093664 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-55599968c8-d5h5x_827c8e52-c812-41e1-9a4b-fd7478baa117/registry/0.log" Jul 16 22:45:23.120146 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:23.120120 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-65cgd_f64092be-22df-4e1c-ab7e-4fcc7afb7ec7/node-ca/0.log" Jul 16 22:45:24.421388 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:24.421361 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54f57477db-cw8lx_2db26749-c5a9-4647-87f0-de77f6ff2f3f/kube-auth-proxy/0.log" Jul 16 22:45:25.341156 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:25.341123 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fbvhc_11bd6b74-116b-4233-a3f5-5b12c68be2f9/serve-healthcheck-canary/0.log" Jul 16 22:45:25.362400 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:25.362243 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-mm8vb" Jul 16 22:45:26.039930 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:26.039904 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7j8zw_41c76e5f-e9ad-4b6b-b8c8-529fa032c570/kube-rbac-proxy/0.log" Jul 16 22:45:26.076855 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:26.076822 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7j8zw_41c76e5f-e9ad-4b6b-b8c8-529fa032c570/exporter/0.log" Jul 16 22:45:26.110741 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:26.110709 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7j8zw_41c76e5f-e9ad-4b6b-b8c8-529fa032c570/extractor/0.log" Jul 16 22:45:29.188319 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:29.188271 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54f57477db-cw8lx_2db26749-c5a9-4647-87f0-de77f6ff2f3f/kube-auth-proxy/0.log" Jul 16 22:45:29.198176 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:29.198151 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-xg9kf_40d4f474-fbc3-4196-bf87-8a3510483609/server/0.log" Jul 16 22:45:29.599493 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:29.599453 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-574666dc8f-7dmxp_f84b0d43-3571-419d-8232-2f7418f2da3a/manager/0.log" Jul 16 22:45:29.622668 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:29.622642 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-574666dc8f-7dmxp_f84b0d43-3571-419d-8232-2f7418f2da3a/copy-manifests/0.log" Jul 16 22:45:30.901088 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:30.901054 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54f57477db-cw8lx_2db26749-c5a9-4647-87f0-de77f6ff2f3f/kube-auth-proxy/0.log" Jul 16 22:45:30.947424 ip-10-0-128-255 kubenswrapper[2585]: I0716 22:45:30.947397 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6bfbbdbb89-9spw5_1ba61693-5313-4873-b93b-ee97d158d3e9/manager/0.log"